问题描述
我遇到了从 Jupyt 运行的 pyspark 的 .toPandas() 方法的奇怪行为。例如,如果我尝试这个:
data = [{"Category": 'Category A',"ID": 1,"Value": 12.40},{"Category": 'Category B',"ID": 2,"Value": 30.10},{"Category": 'Category C',"ID": 3,"Value": 100.01}
]
# Create data frame (where spark is a SparkSession)
df = spark.createDataFrame(data)
df.show()
我能够成功创建 pyspark 数据框。但是,当转换为熊猫时,我得到 IndexError: index is out of bounds:
IndexError Traceback (most recent call last)
<path_to_python>/lib/python3.7/site-packages/IPython/core/formatters.py in __call__(self,obj)
700 type_pprinters=self.type_printers,701 deferred_pprinters=self.deferred_printers)
--> 702 printer.pretty(obj)
703 printer.flush()
704 return stream.getvalue()
<path_to_python>/lib/python3.7/site-packages/IPython/lib/pretty.py in pretty(self,obj)
400 if cls is not object \
401 and callable(cls.__dict__.get('__repr__')):
--> 402 return _repr_pprint(obj,self,cycle)
403
404 return _default_pprint(obj,cycle)
<path_to_python>/lib/python3.7/site-packages/IPython/lib/pretty.py in _repr_pprint(obj,p,cycle)
695 """A pprint that just redirects to the normal repr function."""
696 # Find newlines and replace them with p.break_()
--> 697 output = repr(obj)
698 for idx,output_line in enumerate(output.splitlines()):
699 if idx:
<path_to_python>/lib/python3.7/site-packages/pandas/core/base.py in __repr__(self)
76 Yields Bytestring in Py2,Unicode String in py3.
77 """
---> 78 return str(self)
79
80
<path_to_python>/lib/python3.7/site-packages/pandas/core/base.py in __str__(self)
55
56 if compat.PY3:
---> 57 return self.__unicode__()
58 return self.__bytes__()
59
<path_to_python>/lib/python3.7/site-packages/pandas/core/frame.py in __unicode__(self)
632 width = None
633 self.to_string(buf=buf,max_rows=max_rows,max_cols=max_cols,--> 634 line_width=width,show_dimensions=show_dimensions)
635
636 return buf.getvalue()
<path_to_python>/lib/python3.7/site-packages/pandas/core/frame.py in to_string(self,buf,columns,col_space,header,index,na_rep,formatters,float_format,sparsify,index_names,justify,max_rows,max_cols,show_dimensions,decimal,line_width)
719 decimal=decimal,720 line_width=line_width)
--> 721 formatter.to_string()
722
723 if buf is None:
<path_to_python>/lib/python3.7/site-packages/pandas/io/formats/format.py in to_string(self)
596 else:
597
--> 598 strcols = self._to_str_columns()
599 if self.line_width is None: # no need to wrap around just print
600 # the whole frame
<path_to_python>/lib/python3.7/site-packages/pandas/io/formats/format.py in _to_str_columns(self)
527 str_columns = [[label] for label in self.header]
528 else:
--> 529 str_columns = self._get_formatted_column_labels(frame)
530
531 stringified = []
<path_to_python>/lib/python3.7/site-packages/pandas/io/formats/format.py in _get_formatted_column_labels(self,frame)
770 need_leadsp[x] else x]
771 for i,(col,x) in enumerate(zip(columns,--> 772 fmt_columns))]
773
774 if self.show_row_idx_names:
<path_to_python>/lib/python3.7/site-packages/pandas/io/formats/format.py in <listcomp>(.0)
769 str_columns = [[' ' + x if not self._get_formatter(i) and
770 need_leadsp[x] else x]
--> 771 for i,772 fmt_columns))]
773
<path_to_python>/lib/python3.7/site-packages/pandas/io/formats/format.py in _get_formatter(self,i)
362 else:
363 if is_integer(i) and i not in self.columns:
--> 364 i = self.columns[i]
365 return self.formatters.get(i,None)
366
<path_to_python>/lib/python3.7/site-packages/pandas/core/indexes/base.py in __getitem__(self,key)
3956 if is_scalar(key):
3957 key = com.cast_scalar_indexer(key)
-> 3958 return getitem(key)
3959
3960 if isinstance(key,slice):
IndexError: index 3 is out of bounds for axis 0 with size 3
我不确定问题出在哪里,我用过很多次都没有问题,但这次我尝试了一个新环境,却遇到了这个问题。如果它可以帮助我的配置是:
Python:3.7.6; 熊猫:0.24.2; PySpark:2.4.5
有什么想法吗? 谢谢:)
解决方法
我发现了这个问题。试图最小化代码以重现错误我省略了我添加了熊猫设置:
pd.set_option('display.max_columns',-1)
这导致了与正在转换的数据帧无关的错误。为了解决这个问题,我只是指定了正数的列数或无。