配置单元：将SUM函数应用于“映射”或“数组”列

编程问答 2022-06-11

问题描述

我的表格中有两列，如下所示：

colArray
map<string,array<string>>

和

colMap
map<string,map<string,string>>

这就是典型的行的样子

    colArray                                                        colMap
    {outerKey1: [item1,item2],outerKey2: [item3,item4]}             {outerKey1: {innerKey1:innerValue1,innerKey2:innerValue2},{outerKey2: {innerKey3:innerValue3,innerKey4:innerValue4}}

现在我想对这些列应用一些聚合。像这样的东西

select map(skey,sum(if (size(colArray[skey])>0,1,0)))  totalfromarray from mytable lateral view explode (colArray) ps as skey,svalue
       WHERE dateCol = '2020-08-18';

因此，基本上我想使用一个colArray中存在的键的映射，但是值是一个SUM，如果size(colArray[somwKey])>0我想将其算作1。因此，我的查询具有

map(skey,0)))

我可以对colMap列进行类似的查询

select map(skey,sum(if (size(colMap[skey])>0,0)))  totalfromarray from mytable lateral view explode (colMap) ps as skey,svalue
           WHERE dateCol = '2020-08-18';

但是当我尝试运行上述任何查询时，我都会得到

Caused by: java.lang.IllegalArgumentException: Unable to create serializer "org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer" for class: java.util.concurrent.atomic.atomicreference
    at org.apache.hive.com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:67)
    at org.apache.hive.com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:45)
    at org.apache.hive.com.esotericsoftware.kryo.Kryo.newDefaultSerializer(Kryo.java:380)
    at org.apache.hive.com.esotericsoftware.kryo.Kryo.getDefaultSerializer(Kryo.java:364)
    at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.registerImplicit(DefaultClassResolver.java:74)
    at org.apache.hive.com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:490)
    at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:166)
    at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:133)
    at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670)
    at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClass(SerializationUtilities.java:181)
    at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:118)
    ... 103 more
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.GeneratedConstructorAccessor5.newInstance(UnkNown Source)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at org.apache.hive.com.esotericsoftware.kryo.factories.ReflectionSerializerFactory.makeSerializer(ReflectionSerializerFactory.java:54)
    ... 113 more
Caused by: java.lang.StackOverflowError
    at java.util.HashMap.hash(HashMap.java:339)
    at java.util.HashMap.get(HashMap.java:557)

我在做什么错？我的方法是正确的，还是有更好的方法实现我的目标？

解决方法

暂无找到可以解决该程序问题的有效方法，小编努力寻找整理中！

如果你已经找到好的解决方法，欢迎将解决方案带上本链接一起发送给小编。

小编邮箱:dio#foxmail.com (将#修改为@）

aggregate-functionshivehiveqljavajavasqlsql