问题描述
我是 Oozie 的新手
我们有一个工作流程,我们先调用预处理 Python、Spark 和后处理 Python 作业
如果我们为单个实体调用工作流,它会立即得到处理
但是随着实体数量的不断增加,每个作业的调用开始花费大量时间。 作业的处理仍然很快,但作业的调用需要时间。
以下是我们保留的 oozie 配置:
<property>
<name>oozie.service.CallableQueueService.queue.size</name>
<value>10000</value>
<description>Max callable queue size</description>
</property>
-<property>
<name>oozie.service.SchedulerService.threads</name>
<value>100</value>
<description>The number of threads to be used by the SchedulerService to run deamon tasks.If maxed out,scheduled daemon tasks will be queued up and delayed until threads become available. </description>
</property>
-<property>
<name>oozie.service.CallableQueueService.threads</name>
<value>600</value>
<description>Number of threads used for executing callables</description>
</property>
<property>
<name>oozie.service.CallableQueueService.callable.concurrency</name>
<value>200</value>
<description>Maximum concurrency for a given callable type.Each command is a callable type (submit,start,run,signal,job,jobs,suspend,resume,etc).Each action type is a callable type (Map-Reduce,Pig,SSH,FS,sub-workflow,etc).All commands that use action executors (action-start,action-end,action-kill and action-check) usethe action type as the callable type. </description>
</property>
<property>
<name>oozie.service.coord.normal.default.timeout </name>
<value>120</value>
<description>Default timeout for a coordinator action input check (in minutes) for normal job. -1 means infinite timeout</description>
</property>
-<property>
<name>oozie.action.launcher.mapreduce.job.ubertask.enable</name>
<value>true</value>
</property>
-<property>
<name>oozie.action.shell.launcher.mapreduce.job.ubertask.enable</name>
<value>true</value>
</property>
我们尝试过更改各种值,但没有看到显着的改进。 请支持并提出建议
解决方法
暂无找到可以解决该程序问题的有效方法,小编努力寻找整理中!
如果你已经找到好的解决方法,欢迎将解决方案带上本链接一起发送给小编。
小编邮箱:dio#foxmail.com (将#修改为@)