问题描述
我正在尝试使用PandasSchema验证我的DataFrame库伦。我被困在验证某些列,例如:
1.ip_address-应该包含以下格式的IP地址:1.1.1.1;如果其他任何值会引发错误,则应为null。 2.initial_date-格式yyyy-mm-dd h:m:s或mm-dd-yyyy h:m:s等 3.customertype应该在['type1','type2','type3']中,否则会引发错误。 4.客户满意=是/否或空白 5.customerid不得超过5个字符,例如cus01,cus02 6.time应该采用%:%:格式或h:m:s格式,其他任何情况都会引发异常。
from pandas_schema import Column,Schema
def check_string(sr):
try:
str(sr)
except InvalidOperation:
return False
return True
def check_datetime(self,dec):
try:
datetime.datetime.strptime(dec,self.date_format)
return True
except:
return False
def check_int(num):
try:
int(num)
except ValueError:
return False
return True
string_validation=[CustomElementValidation(lambda x: check_string(x).str.len()>5,'Field Correct')]
int_validation = [CustomElementValidation(lambda i: check_int(i),'is not integer')]
contain_validation = [CustomElementValidation(lambda y: check_string(y) not in['type1','type2','type3'],'Filed is correct')]
date_time_validation=[CustomElementValidation(lambda dt: check_datetime(dt).strptime('%m/%d/%Y %H:%M %p'),'is not a date
time')]
null_validation = [CustomElementValidation(lambda d: d is not np.nan,'this field cannot be null')]
schema = Schema([
Column('CompanyID',string_validation + null_validation),Column('initialdate',date_time_validation),Column('customertype',contain_validation),Column('ip',string_validation),Column('customersatisfied',string_validation)])
errors = schema.validate(combined_df)
errors_index_rows = [e.row for e in errors]
pd.DataFrame({'col':errors}).to_csv('errors.csv')
解决方法
我刚刚查看了PandasShema的文档,并且大多数(如果不是全部的话)正在寻找现成的功能。看看:
作为快速解决您的问题的方法,应该可以执行以下操作:
from pandas_schema.validation import (
InListValidation,IsDtypeValidation,DateFormatValidation,MatchesPatternValidation
)
schema = Schema([
# Match a string of length between 1 and 5
Column('CompanyID',[MatchesPatternValidation(r".{1,5}")]),# Match a date-like string of ISO 8601 format (https://www.iso.org/iso-8601-date-and-time-format.html)
Column('initialdate',[DateFormatValidation("%Y-%m-%d %H:%M:%S")],allow_empty=True),# Match only strings in the following list
Column('customertype',[InListValidation(["type1","type2","type3"])]),# Match an IP address RegEx (https://www.oreilly.com/library/view/regular-expressions-cookbook/9780596802837/ch07s16.html)
Column('ip',[MatchesPatternValidation(r"(?:[0-9]{1,3}\.){3}[0-9]{1,3}")]),# Match only strings in the following list
Column('customersatisfied',[InListValidation(["yes","no"])],allow_empty=True)
])