Pyspark：to_date 和 unix_timestamp 为其他有效值的某些记录返回 null

问题描述

上面的命令为少数记录提供了空值。 任何人都可以告诉这可能是什么原因。我曾尝试使用 to_date 和 unix_timestamp 函数，但两者都给出相同的结果（少数值为 null）。

date_cols.withColumn('date_timestamp',unix_timestamp('Date',"M/dd/y HH:mm:ss a")).show(truncate=False)


|----------------------|----------------------|--------------|
|Date                  |Updated On            |date_timestamp|
|----------------------|----------------------|--------------|
|02/05/2018 01:10:00 AM|02/12/2018 03:49:14 PM|1517793000    |
|01/23/2018 09:00:00 AM|02/12/2018 03:49:14 PM|1516698000    |
|01/31/2018 10:12:00 AM|02/12/2018 03:49:14 PM|1517393520    |
|02/01/2018 02:00:00 PM|02/12/2018 03:49:14 PM|null          |
|01/27/2018 09:00:00 PM|02/12/2018 03:49:14 PM|null          |
|02/02/2018 12:00:00 PM|02/12/2018 03:49:14 PM|1517572800    |
|01/28/2018 07:00:00 PM|02/12/2018 03:49:14 PM|null          |
|01/10/2018 04:45:00 PM|02/12/2018 03:49:14 PM|null          |
|01/03/2018 03:45:00 PM|02/12/2018 03:49:14 PM|null          |
|02/04/2018 04:00:00 PM|02/12/2018 03:49:14 PM|null          |
|01/27/2018 07:00:00 AM|02/12/2018 03:49:14 PM|1517036400    |
|02/05/2018 12:00:00 AM|02/12/2018 03:49:14 PM|null          |
|01/21/2018 02:00:00 AM|02/12/2018 03:49:14 PM|1516500000    |
|02/02/2018 12:45:00 AM|02/12/2018 03:49:14 PM|null          |
|01/25/2018 01:14:00 PM|02/12/2018 03:49:14 PM|null          |
|02/05/2018 05:00:00 PM|02/12/2018 03:49:14 PM|null          |
|01/15/2018 12:00:00 AM|02/12/2018 03:49:14 PM|null          |
|02/05/2018 07:30:00 PM|02/12/2018 03:49:14 PM|null          |
|02/02/2018 08:30:00 PM|02/12/2018 03:49:14 PM|null          |
|02/03/2018 06:00:00 PM|02/12/2018 03:49:14 PM|null          |
+----------------------+----------------------+--------------+

解决方法

实际上，我犯了一个小错误，在查看了 [doc][1]

它得到了解决，所以在这里发布，它可能会帮助某人。 [1]：https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html

在格式化时必须使用 hh 而不是 HH。由于hh 代表clock-hour-of-am-pm (1-12) 而HH 代表一天中的小时（ 0-23)

date_cols.withColumn('date_timestamp',unix_timestamp('Date',"MM/dd/y hh:mm:ss a").cast(TimestampType())).show(truncate=False)

|----------------------|----------------------|-------------------|
|Date                  |Updated On            |date_timestamp     |
|----------------------|----------------------|-------------------|
|02/05/2018 01:10:00 AM|02/12/2018 03:49:14 PM|2018-02-05 01:10:00|
|01/23/2018 09:00:00 AM|02/12/2018 03:49:14 PM|2018-01-23 09:00:00|
|01/31/2018 10:12:00 AM|02/12/2018 03:49:14 PM|2018-01-31 10:12:00|
|02/01/2018 02:00:00 PM|02/12/2018 03:49:14 PM|2018-02-01 14:00:00|
|01/27/2018 09:00:00 PM|02/12/2018 03:49:14 PM|2018-01-27 21:00:00|
|02/02/2018 12:00:00 PM|02/12/2018 03:49:14 PM|2018-02-02 12:00:00|
|01/28/2018 07:00:00 PM|02/12/2018 03:49:14 PM|2018-01-28 19:00:00|
|01/10/2018 04:45:00 PM|02/12/2018 03:49:14 PM|2018-01-10 16:45:00|
|01/03/2018 03:45:00 PM|02/12/2018 03:49:14 PM|2018-01-03 15:45:00|
|02/04/2018 04:00:00 PM|02/12/2018 03:49:14 PM|2018-02-04 16:00:00|
|01/27/2018 07:00:00 AM|02/12/2018 03:49:14 PM|2018-01-27 07:00:00|
|02/05/2018 12:00:00 AM|02/12/2018 03:49:14 PM|2018-02-05 00:00:00|
|01/21/2018 02:00:00 AM|02/12/2018 03:49:14 PM|2018-01-21 02:00:00|
|02/02/2018 12:45:00 AM|02/12/2018 03:49:14 PM|2018-02-02 00:45:00|
|01/25/2018 01:14:00 PM|02/12/2018 03:49:14 PM|2018-01-25 13:14:00|
|02/05/2018 05:00:00 PM|02/12/2018 03:49:14 PM|2018-02-05 17:00:00|
|01/15/2018 12:00:00 AM|02/12/2018 03:49:14 PM|2018-01-15 00:00:00|
|02/05/2018 07:30:00 PM|02/12/2018 03:49:14 PM|2018-02-05 19:30:00|
|02/02/2018 08:30:00 PM|02/12/2018 03:49:14 PM|2018-02-02 20:30:00|
|02/03/2018 06:00:00 PM|02/12/2018 03:49:14 PM|2018-02-03 18:00:00|
|----------------------|----------------------|-------------------|