moryStore: Block broadcast_152_piece0 stored as bytes in memory (estimated size 10.0 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO TorrentBroadcast: Reading broadcast variable 152 took 3 ms
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_152 stored as values in memory (estimated size 10.4 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO Executor: Finished task 0.0 in stage 164.0 (TID 251). 14002 bytes result sent to driver
23/09/24 21:18:27 INFO CoarseGrainedExecutorBackend: Got assigned task 252
23/09/24 21:18:27 INFO Executor: Running task 0.0 in stage 165.0 (TID 252)
23/09/24 21:18:27 INFO TorrentBroadcast: Started reading broadcast variable 155 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_155_piece0 stored as bytes in memory (estimated size 53.2 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO TorrentBroadcast: Reading broadcast variable 155 took 3 ms
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_155 stored as values in memory (estimated size 195.7 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO BlockManager: Found block rdd_547_0 locally
23/09/24 21:18:27 INFO TorrentBroadcast: Started reading broadcast variable 154 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_154_piece0 stored as bytes in memory (estimated size 10.0 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO TorrentBroadcast: Reading broadcast variable 154 took 2 ms
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_154 stored as values in memory (estimated size 10.4 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO Executor: Finished task 0.0 in stage 165.0 (TID 252). 14002 bytes result sent to driver
23/09/24 21:18:27 INFO CoarseGrainedExecutorBackend: Got assigned task 253
23/09/24 21:18:27 INFO Executor: Running task 0.0 in stage 166.0 (TID 253)
23/09/24 21:18:27 INFO TorrentBroadcast: Started reading broadcast variable 157 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_157_piece0 stored as bytes in memory (estimated size 53.2 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO TorrentBroadcast: Reading broadcast variable 157 took 3 ms
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_157 stored as values in memory (estimated size 195.7 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO BlockManager: Found block rdd_547_0 locally
23/09/24 21:18:27 INFO TorrentBroadcast: Started reading broadcast variable 156 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_156_piece0 stored as bytes in memory (estimated size 10.0 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO TorrentBroadcast: Reading broadcast variable 156 took 2 ms
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_156 stored as values in memory (estimated size 10.4 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO Executor: Finished task 0.0 in stage 166.0 (TID 253). 14002 bytes result sent to driver
23/09/24 21:18:27 INFO CoarseGrainedExecutorBackend: Got assigned task 254
23/09/24 21:18:27 INFO Executor: Running task 0.0 in stage 167.0 (TID 254)
23/09/24 21:18:27 INFO TorrentBroadcast: Started reading broadcast variable 159 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_159_piece0 stored as bytes in memory (estimated size 53.2 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO TorrentBroadcast: Reading broadcast variable 159 took 3 ms
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_159 stored as values in memory (estimated size 195.7 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO BlockManager: Found block rdd_547_0 locally
23/09/24 21:18:27 INFO TorrentBroadcast: Started reading broadcast variable 158 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_158_piece0 stored as bytes in memory (estimated size 10.0 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO TorrentBroadcast: Reading broadcast variable 158 took 2 ms
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_158 stored as values in memory (estimated size 10.4 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO Executor: Finished task 0.0 in stage 167.0 (TID 254). 14002 bytes result sent to driver
23/09/24 21:18:27 INFO CoarseGrainedExecutorBackend: Got assigned task 255
23/09/24 21:18:27 INFO Executor: Running task 0.0 in stage 168.0 (TID 255)
23/09/24 21:18:27 INFO TorrentBroadcast: Started reading broadcast variable 161 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_161_piece0 stored as bytes in memory (estimated size 53.2 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO TorrentBroadcast: Reading broadcast variable 161 took 3 ms
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_161 stored as values in memory (estimated size 195.7 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO BlockManager: Found block rdd_547_0 locally
23/09/24 21:18:27 INFO TorrentBroadcast: Started reading broadcast variable 160 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_160_piece0 stored as bytes in memory (estimated size 10.1 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO TorrentBroadcast: Reading broadcast variable 160 took 3 ms
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_160 stored as values in memory (estimated size 10.4 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO Executor: Finished task 0.0 in stage 168.0 (TID 255). 14045 bytes result sent to driver
23/09/24 21:18:27 INFO CoarseGrainedExecutorBackend: Got assigned task 256
23/09/24 21:18:27 INFO Executor: Running task 0.0 in stage 169.0 (TID 256)
23/09/24 21:18:27 INFO TorrentBroadcast: Started reading broadcast variable 163 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_163_piece0 stored as bytes in memory (estimated size 53.2 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO TorrentBroadcast: Reading broadcast variable 163 took 2 ms
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_163 stored as values in memory (estimated size 195.7 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO BlockManager: Found block rdd_547_0 locally
23/09/24 21:18:27 INFO TorrentBroadcast: Started reading broadcast variable 162 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_162_piece0 stored as bytes in memory (estimated size 10.1 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO TorrentBroadcast: Reading broadcast variable 162 took 2 ms
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_162 stored as values in memory (estimated size 10.4 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO Executor: Finished task 0.0 in stage 169.0 (TID 256). 14002 bytes result sent to driver
23/09/24 21:18:27 INFO CoarseGrainedExecutorBackend: Got assigned task 257
23/09/24 21:18:27 INFO Executor: Running task 0.0 in stage 170.0 (TID 257)
23/09/24 21:18:27 INFO TorrentBroadcast: Started reading broadcast variable 165 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_165_piece0 stored as bytes in memory (estimated size 53.2 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO TorrentBroadcast: Reading broadcast variable 165 took 3 ms
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_165 stored as values in memory (estimated size 195.7 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO BlockManager: Found block rdd_547_0 locally
23/09/24 21:18:27 INFO TorrentBroadcast: Started reading broadcast variable 164 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_164_piece0 stored as bytes in memory (estimated size 10.0 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO TorrentBroadcast: Reading broadcast variable 164 took 3 ms
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_164 stored as values in memory (estimated size 10.4 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO Executor: Finished task 0.0 in stage 170.0 (TID 257). 14002 bytes result sent to driver
23/09/24 21:18:27 INFO CoarseGrainedExecutorBackend: Got assigned task 258
23/09/24 21:18:27 INFO Executor: Running task 0.0 in stage 171.0 (TID 258)
23/09/24 21:18:27 INFO TorrentBroadcast: Started reading broadcast variable 167 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_167_piece0 stored as bytes in memory (estimated size 53.2 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO TorrentBroadcast: Reading broadcast variable 167 took 2 ms
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_167 stored as values in memory (estimated size 195.7 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO BlockManager: Found block rdd_547_0 locally
23/09/24 21:18:27 INFO TorrentBroadcast: Started reading broadcast variable 166 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_166_piece0 stored as bytes in memory (estimated size 10.1 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO TorrentBroadcast: Reading broadcast variable 166 took 2 ms
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_166 stored as values in memory (estimated size 10.4 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO Executor: Finished task 0.0 in stage 171.0 (TID 258). 14002 bytes result sent to driver
23/09/24 21:18:27 INFO CoarseGrainedExecutorBackend: Got assigned task 259
23/09/24 21:18:27 INFO Executor: Running task 0.0 in stage 172.0 (TID 259)
23/09/24 21:18:27 INFO TorrentBroadcast: Started reading broadcast variable 169 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_169_piece0 stored as bytes in memory (estimated size 53.2 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO TorrentBroadcast: Reading broadcast variable 169 took 3 ms
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_169 stored as values in memory (estimated size 195.7 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO BlockManager: Found block rdd_547_0 locally
23/09/24 21:18:27 INFO TorrentBroadcast: Started reading broadcast variable 168 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_168_piece0 stored as bytes in memory (estimated size 10.0 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO TorrentBroadcast: Reading broadcast variable 168 took 2 ms
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_168 stored as values in memory (estimated size 10.4 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO Executor: Finished task 0.0 in stage 172.0 (TID 259). 14002 bytes result sent to driver
23/09/24 21:18:27 INFO CoarseGrainedExecutorBackend: Got assigned task 260
23/09/24 21:18:27 INFO Executor: Running task 0.0 in stage 173.0 (TID 260)
23/09/24 21:18:27 INFO TorrentBroadcast: Started reading broadcast variable 171 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_171_piece0 stored as bytes in memory (estimated size 53.2 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO TorrentBroadcast: Reading broadcast variable 171 took 2 ms
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_171 stored as values in memory (estimated size 195.7 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO BlockManager: Found block rdd_547_0 locally
23/09/24 21:18:27 INFO TorrentBroadcast: Started reading broadcast variable 170 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_170_piece0 stored as bytes in memory (estimated size 10.0 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO TorrentBroadcast: Reading broadcast variable 170 took 2 ms
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_170 stored as values in memory (estimated size 10.4 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO Executor: Finished task 0.0 in stage 173.0 (TID 260). 14002 bytes result sent to driver
23/09/24 21:18:27 INFO CoarseGrainedExecutorBackend: Got assigned task 261
23/09/24 21:18:27 INFO Executor: Running task 0.0 in stage 174.0 (TID 261)
23/09/24 21:18:27 INFO TorrentBroadcast: Started reading broadcast variable 173 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_173_piece0 stored as bytes in memory (estimated size 53.2 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO TorrentBroadcast: Reading broadcast variable 173 took 2 ms
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_173 stored as values in memory (estimated size 195.7 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO BlockManager: Found block rdd_547_0 locally
23/09/24 21:18:27 INFO TorrentBroadcast: Started reading broadcast variable 172 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_172_piece0 stored as bytes in memory (estimated size 10.1 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO TorrentBroadcast: Reading broadcast variable 172 took 2 ms
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_172 stored as values in memory (estimated size 10.4 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO Executor: Finished task 0.0 in stage 174.0 (TID 261). 14002 bytes result sent to driver
23/09/24 21:18:27 INFO CoarseGrainedExecutorBackend: Got assigned task 262
23/09/24 21:18:27 INFO Executor: Running task 0.0 in stage 175.0 (TID 262)
23/09/24 21:18:27 INFO TorrentBroadcast: Started reading broadcast variable 175 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_175_piece0 stored as bytes in memory (estimated size 53.2 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO TorrentBroadcast: Reading broadcast variable 175 took 2 ms
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_175 stored as values in memory (estimated size 195.7 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO BlockManager: Found block rdd_547_0 locally
23/09/24 21:18:27 INFO TorrentBroadcast: Started reading broadcast variable 174 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_174_piece0 stored as bytes in memory (estimated size 10.0 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO TorrentBroadcast: Reading broadcast variable 174 took 1 ms
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_174 stored as values in memory (estimated size 10.4 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO Executor: Finished task 0.0 in stage 175.0 (TID 262). 14002 bytes result sent to driver
23/09/24 21:18:27 INFO CoarseGrainedExecutorBackend: Got assigned task 263
23/09/24 21:18:27 INFO Executor: Running task 0.0 in stage 176.0 (TID 263)
23/09/24 21:18:27 INFO TorrentBroadcast: Started reading broadcast variable 177 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_177_piece0 stored as bytes in memory (estimated size 53.2 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO TorrentBroadcast: Reading broadcast variable 177 took 2 ms
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_177 stored as values in memory (estimated size 195.7 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO BlockManager: Found block rdd_547_0 locally
23/09/24 21:18:27 INFO TorrentBroadcast: Started reading broadcast variable 176 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_176_piece0 stored as bytes in memory (estimated size 10.0 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO TorrentBroadcast: Reading broadcast variable 176 took 1 ms
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_176 stored as values in memory (estimated size 10.4 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO Executor: Finished task 0.0 in stage 176.0 (TID 263). 14002 bytes result sent to driver
23/09/24 21:18:27 INFO CoarseGrainedExecutorBackend: Got assigned task 264
23/09/24 21:18:27 INFO Executor: Running task 0.0 in stage 177.0 (TID 264)
23/09/24 21:18:27 INFO TorrentBroadcast: Started reading broadcast variable 179 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_179_piece0 stored as bytes in memory (estimated size 53.2 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO TorrentBroadcast: Reading broadcast variable 179 took 2 ms
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_179 stored as values in memory (estimated size 195.7 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO BlockManager: Found block rdd_547_0 locally
23/09/24 21:18:27 INFO TorrentBroadcast: Started reading broadcast variable 178 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_178_piece0 stored as bytes in memory (estimated size 10.1 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO TorrentBroadcast: Reading broadcast variable 178 took 1 ms
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_178 stored as values in memory (estimated size 10.4 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO Executor: Finished task 0.0 in stage 177.0 (TID 264). 14002 bytes result sent to driver
23/09/24 21:18:27 INFO CoarseGrainedExecutorBackend: Got assigned task 265
23/09/24 21:18:27 INFO Executor: Running task 0.0 in stage 178.0 (TID 265)
23/09/24 21:18:27 INFO TorrentBroadcast: Started reading broadcast variable 181 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_181_piece0 stored as bytes in memory (estimated size 53.2 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO TorrentBroadcast: Reading broadcast variable 181 took 2 ms
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_181 stored as values in memory (estimated size 195.7 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO BlockManager: Found block rdd_547_0 locally
23/09/24 21:18:27 INFO TorrentBroadcast: Started reading broadcast variable 180 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_180_piece0 stored as bytes in memory (estimated size 10.1 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO TorrentBroadcast: Reading broadcast variable 180 took 2 ms
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_180 stored as values in memory (estimated size 10.4 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO Executor: Finished task 0.0 in stage 178.0 (TID 265). 14002 bytes result sent to driver
23/09/24 21:18:27 INFO CoarseGrainedExecutorBackend: Got assigned task 266
23/09/24 21:18:27 INFO Executor: Running task 0.0 in stage 179.0 (TID 266)
23/09/24 21:18:27 INFO TorrentBroadcast: Started reading broadcast variable 183 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_183_piece0 stored as bytes in memory (estimated size 53.2 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO TorrentBroadcast: Reading broadcast variable 183 took 3 ms
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_183 stored as values in memory (estimated size 195.7 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO BlockManager: Found block rdd_547_0 locally
23/09/24 21:18:27 INFO TorrentBroadcast: Started reading broadcast variable 182 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_182_piece0 stored as bytes in memory (estimated size 10.0 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO TorrentBroadcast: Reading broadcast variable 182 took 3 ms
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_182 stored as values in memory (estimated size 10.4 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO Executor: Finished task 0.0 in stage 179.0 (TID 266). 14002 bytes result sent to driver
23/09/24 21:18:27 INFO CoarseGrainedExecutorBackend: Got assigned task 267
23/09/24 21:18:27 INFO Executor: Running task 0.0 in stage 180.0 (TID 267)
23/09/24 21:18:27 INFO TorrentBroadcast: Started reading broadcast variable 185 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_185_piece0 stored as bytes in memory (estimated size 53.2 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO TorrentBroadcast: Reading broadcast variable 185 took 3 ms
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_185 stored as values in memory (estimated size 195.7 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO BlockManager: Found block rdd_547_0 locally
23/09/24 21:18:27 INFO TorrentBroadcast: Started reading broadcast variable 184 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_184_piece0 stored as bytes in memory (estimated size 10.0 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO TorrentBroadcast: Reading broadcast variable 184 took 3 ms
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_184 stored as values in memory (estimated size 10.4 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO Executor: Finished task 0.0 in stage 180.0 (TID 267). 14002 bytes result sent to driver
23/09/24 21:18:27 INFO CoarseGrainedExecutorBackend: Got assigned task 268
23/09/24 21:18:27 INFO Executor: Running task 0.0 in stage 181.0 (TID 268)
23/09/24 21:18:27 INFO TorrentBroadcast: Started reading broadcast variable 187 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_187_piece0 stored as bytes in memory (estimated size 53.2 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO TorrentBroadcast: Reading broadcast variable 187 took 3 ms
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_187 stored as values in memory (estimated size 195.7 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO BlockManager: Found block rdd_547_0 locally
23/09/24 21:18:27 INFO TorrentBroadcast: Started reading broadcast variable 186 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_186_piece0 stored as bytes in memory (estimated size 10.0 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO TorrentBroadcast: Reading broadcast variable 186 took 3 ms
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_186 stored as values in memory (estimated size 10.4 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO Executor: Finished task 0.0 in stage 181.0 (TID 268). 14002 bytes result sent to driver
23/09/24 21:18:27 INFO CoarseGrainedExecutorBackend: Got assigned task 269
23/09/24 21:18:27 INFO Executor: Running task 0.0 in stage 182.0 (TID 269)
23/09/24 21:18:27 INFO TorrentBroadcast: Started reading broadcast variable 189 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_189_piece0 stored as bytes in memory (estimated size 53.2 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO TorrentBroadcast: Reading broadcast variable 189 took 2 ms
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_189 stored as values in memory (estimated size 195.7 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO BlockManager: Found block rdd_547_0 locally
23/09/24 21:18:27 INFO TorrentBroadcast: Started reading broadcast variable 188 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_188_piece0 stored as bytes in memory (estimated size 10.0 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO TorrentBroadcast: Reading broadcast variable 188 took 2 ms
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_188 stored as values in memory (estimated size 10.4 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO Executor: Finished task 0.0 in stage 182.0 (TID 269). 14001 bytes result sent to driver
23/09/24 21:18:27 INFO CoarseGrainedExecutorBackend: Got assigned task 270
23/09/24 21:18:27 INFO Executor: Running task 0.0 in stage 183.0 (TID 270)
23/09/24 21:18:27 INFO TorrentBroadcast: Started reading broadcast variable 191 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_191_piece0 stored as bytes in memory (estimated size 53.2 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO TorrentBroadcast: Reading broadcast variable 191 took 2 ms
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_191 stored as values in memory (estimated size 195.7 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO BlockManager: Found block rdd_547_0 locally
23/09/24 21:18:27 INFO TorrentBroadcast: Started reading broadcast variable 190 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_190_piece0 stored as bytes in memory (estimated size 10.0 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO TorrentBroadcast: Reading broadcast variable 190 took 2 ms
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_190 stored as values in memory (estimated size 10.4 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO Executor: Finished task 0.0 in stage 183.0 (TID 270). 14002 bytes result sent to driver
23/09/24 21:18:27 INFO CoarseGrainedExecutorBackend: Got assigned task 271
23/09/24 21:18:27 INFO Executor: Running task 0.0 in stage 184.0 (TID 271)
23/09/24 21:18:27 INFO TorrentBroadcast: Started reading broadcast variable 193 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_193_piece0 stored as bytes in memory (estimated size 53.2 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO TorrentBroadcast: Reading broadcast variable 193 took 2 ms
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_193 stored as values in memory (estimated size 195.7 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO BlockManager: Found block rdd_547_0 locally
23/09/24 21:18:27 INFO TorrentBroadcast: Started reading broadcast variable 192 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_192_piece0 stored as bytes in memory (estimated size 10.1 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO TorrentBroadcast: Reading broadcast variable 192 took 2 ms
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_192 stored as values in memory (estimated size 10.4 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO Executor: Finished task 0.0 in stage 184.0 (TID 271). 14001 bytes result sent to driver
23/09/24 21:18:27 INFO CoarseGrainedExecutorBackend: Got assigned task 272
23/09/24 21:18:27 INFO Executor: Running task 0.0 in stage 185.0 (TID 272)
23/09/24 21:18:27 INFO TorrentBroadcast: Started reading broadcast variable 195 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_195_piece0 stored as bytes in memory (estimated size 53.2 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO TorrentBroadcast: Reading broadcast variable 195 took 2 ms
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_195 stored as values in memory (estimated size 195.7 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO BlockManager: Found block rdd_547_0 locally
23/09/24 21:18:27 INFO TorrentBroadcast: Started reading broadcast variable 194 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_194_piece0 stored as bytes in memory (estimated size 10.0 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO TorrentBroadcast: Reading broadcast variable 194 took 2 ms
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_194 stored as values in memory (estimated size 10.4 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO Executor: Finished task 0.0 in stage 185.0 (TID 272). 14002 bytes result sent to driver
23/09/24 21:18:27 INFO CoarseGrainedExecutorBackend: Got assigned task 273
23/09/24 21:18:27 INFO Executor: Running task 0.0 in stage 186.0 (TID 273)
23/09/24 21:18:27 INFO TorrentBroadcast: Started reading broadcast variable 197 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_197_piece0 stored as bytes in memory (estimated size 53.2 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO TorrentBroadcast: Reading broadcast variable 197 took 3 ms
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_197 stored as values in memory (estimated size 195.7 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO BlockManager: Found block rdd_547_0 locally
23/09/24 21:18:27 INFO TorrentBroadcast: Started reading broadcast variable 196 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_196_piece0 stored as bytes in memory (estimated size 10.0 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO TorrentBroadcast: Reading broadcast variable 196 took 7 ms
23/09/24 21:18:27 INFO MemoryStore: Block broadcast_196 stored as values in memory (estimated size 10.4 KiB, free 5.0 GiB)
23/09/24 21:18:27 INFO Executor: Finished task 0.0 in stage 186.0 (TID 273). 14002 bytes result sent to driver
23/09/24 21:18:27 INFO BlockManager: Removing RDD 547
23/09/24 21:18:28 INFO BlockManager: Removing RDD 547
23/09/24 21:18:28 INFO CoarseGrainedExecutorBackend: Got assigned task 274
23/09/24 21:18:28 INFO CoarseGrainedExecutorBackend: Got assigned task 277
23/09/24 21:18:28 INFO Executor: Running task 0.0 in stage 188.0 (TID 274)
23/09/24 21:18:28 INFO Executor: Running task 3.0 in stage 188.0 (TID 277)
23/09/24 21:18:28 INFO TorrentBroadcast: Started reading broadcast variable 198 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:28 INFO MemoryStore: Block broadcast_198_piece0 stored as bytes in memory (estimated size 17.8 KiB, free 5.0 GiB)
23/09/24 21:18:28 INFO TorrentBroadcast: Reading broadcast variable 198 took 3 ms
23/09/24 21:18:28 INFO MemoryStore: Block broadcast_198 stored as values in memory (estimated size 63.4 KiB, free 5.0 GiB)
23/09/24 21:18:28 INFO BlockManager: Found block rdd_88_0 locally
23/09/24 21:18:28 INFO BlockManager: Found block rdd_88_3 locally
23/09/24 21:18:28 INFO CodeGenerator: Code generated in 24.807103 ms
23/09/24 21:18:28 INFO CodeGenerator: Code generated in 16.76524 ms
23/09/24 21:18:28 INFO Executor: Finished task 0.0 in stage 188.0 (TID 274). 2574 bytes result sent to driver
23/09/24 21:18:28 INFO Executor: Finished task 3.0 in stage 188.0 (TID 277). 2574 bytes result sent to driver
23/09/24 21:18:28 INFO CoarseGrainedExecutorBackend: Got assigned task 278
23/09/24 21:18:28 INFO CoarseGrainedExecutorBackend: Got assigned task 281
23/09/24 21:18:28 INFO Executor: Running task 0.0 in stage 189.0 (TID 278)
23/09/24 21:18:28 INFO Executor: Running task 3.0 in stage 189.0 (TID 281)
23/09/24 21:18:28 INFO MapOutputTrackerWorker: Updating epoch to 44 and clearing cache
23/09/24 21:18:28 INFO TorrentBroadcast: Started reading broadcast variable 199 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:28 INFO MemoryStore: Block broadcast_199_piece0 stored as bytes in memory (estimated size 7.9 KiB, free 5.0 GiB)
23/09/24 21:18:28 INFO TorrentBroadcast: Reading broadcast variable 199 took 10 ms
23/09/24 21:18:28 INFO MemoryStore: Block broadcast_199 stored as values in memory (estimated size 19.4 KiB, free 5.0 GiB)
23/09/24 21:18:28 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 43, fetching them
23/09/24 21:18:28 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:41787)
23/09/24 21:18:28 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 43, fetching them
23/09/24 21:18:28 INFO MapOutputTrackerWorker: Got the output locations
23/09/24 21:18:28 INFO ShuffleBlockFetcherIterator: Getting 4 (2.8 KiB) non-empty blocks including 2 (1461.0 B) local and 2 (1456.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 21:18:28 INFO ShuffleBlockFetcherIterator: Getting 2 (721.0 B) non-empty blocks including 2 (721.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 21:18:28 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 21:18:28 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 21:18:28 INFO MemoryStore: Block rdd_610_0 stored as values in memory (estimated size 2.7 KiB, free 5.0 GiB)
23/09/24 21:18:28 INFO MemoryStore: Block rdd_610_3 stored as values in memory (estimated size 4.2 KiB, free 5.0 GiB)
23/09/24 21:18:28 INFO CodeGenerator: Code generated in 11.383427 ms
23/09/24 21:18:28 INFO CodeGenerator: Code generated in 7.11855 ms
23/09/24 21:18:28 INFO Executor: Finished task 0.0 in stage 189.0 (TID 278). 2848 bytes result sent to driver
23/09/24 21:18:28 INFO Executor: Finished task 3.0 in stage 189.0 (TID 281). 4372 bytes result sent to driver
23/09/24 21:18:28 INFO CoarseGrainedExecutorBackend: Got assigned task 282
23/09/24 21:18:28 INFO CoarseGrainedExecutorBackend: Got assigned task 285
23/09/24 21:18:28 INFO Executor: Running task 0.0 in stage 192.0 (TID 282)
23/09/24 21:18:28 INFO Executor: Running task 3.0 in stage 192.0 (TID 285)
23/09/24 21:18:28 INFO TorrentBroadcast: Started reading broadcast variable 201 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:28 INFO MemoryStore: Block broadcast_201_piece0 stored as bytes in memory (estimated size 51.3 KiB, free 5.0 GiB)
23/09/24 21:18:28 INFO TorrentBroadcast: Reading broadcast variable 201 took 5 ms
23/09/24 21:18:28 INFO MemoryStore: Block broadcast_201 stored as values in memory (estimated size 139.3 KiB, free 5.0 GiB)
23/09/24 21:18:29 INFO BlockManager: Found block rdd_610_3 locally
23/09/24 21:18:29 INFO BlockManager: Found block rdd_610_0 locally
23/09/24 21:18:29 INFO CodeGenerator: Code generated in 9.352486 ms
23/09/24 21:18:29 INFO CodeGenerator: Code generated in 11.508021 ms
23/09/24 21:18:29 INFO TorrentBroadcast: Started reading broadcast variable 200 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:29 INFO MemoryStore: Block broadcast_200_piece0 stored as bytes in memory (estimated size 3.9 KiB, free 5.0 GiB)
23/09/24 21:18:29 INFO TorrentBroadcast: Reading broadcast variable 200 took 3 ms
23/09/24 21:18:29 INFO MemoryStore: Block broadcast_200 stored as values in memory (estimated size 7.9 KiB, free 5.0 GiB)
23/09/24 21:18:29 INFO MemoryStore: Block rdd_621_0 stored as values in memory (estimated size 4.6 KiB, free 5.0 GiB)
23/09/24 21:18:29 INFO MemoryStore: Block rdd_621_3 stored as values in memory (estimated size 7.2 KiB, free 5.0 GiB)
23/09/24 21:18:29 INFO CodeGenerator: Code generated in 9.61944 ms
23/09/24 21:18:29 INFO CodeGenerator: Code generated in 16.829251 ms
23/09/24 21:18:29 INFO MemoryStore: Block rdd_627_3 stored as values in memory (estimated size 7.6 KiB, free 5.0 GiB)
23/09/24 21:18:29 INFO MemoryStore: Block rdd_627_0 stored as values in memory (estimated size 4.8 KiB, free 5.0 GiB)
23/09/24 21:18:29 INFO CodeGenerator: Code generated in 8.451965 ms
23/09/24 21:18:29 INFO CodeGenerator: Code generated in 12.809165 ms
23/09/24 21:18:29 INFO CodeGenerator: Code generated in 9.744494 ms
23/09/24 21:18:29 INFO CodeGenerator: Code generated in 5.536074 ms
23/09/24 21:18:29 INFO Executor: Finished task 0.0 in stage 192.0 (TID 282). 2833 bytes result sent to driver
23/09/24 21:18:29 INFO Executor: Finished task 3.0 in stage 192.0 (TID 285). 2968 bytes result sent to driver
23/09/24 21:18:30 INFO CoarseGrainedExecutorBackend: Got assigned task 288
23/09/24 21:18:30 INFO CoarseGrainedExecutorBackend: Got assigned task 289
23/09/24 21:18:30 INFO Executor: Running task 0.0 in stage 195.0 (TID 288)
23/09/24 21:18:30 INFO Executor: Running task 3.0 in stage 195.0 (TID 289)
23/09/24 21:18:30 INFO TorrentBroadcast: Started reading broadcast variable 202 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:30 INFO MemoryStore: Block broadcast_202_piece0 stored as bytes in memory (estimated size 51.3 KiB, free 5.0 GiB)
23/09/24 21:18:30 INFO TorrentBroadcast: Reading broadcast variable 202 took 3 ms
23/09/24 21:18:30 INFO MemoryStore: Block broadcast_202 stored as values in memory (estimated size 139.3 KiB, free 5.0 GiB)
23/09/24 21:18:30 INFO BlockManager: Found block rdd_627_3 locally
23/09/24 21:18:30 INFO BlockManager: Found block rdd_627_0 locally
23/09/24 21:18:30 INFO CodeGenerator: Code generated in 8.96304 ms
23/09/24 21:18:30 INFO CodeGenerator: Code generated in 7.948316 ms
23/09/24 21:18:30 INFO Executor: Finished task 3.0 in stage 195.0 (TID 289). 3036 bytes result sent to driver
23/09/24 21:18:30 INFO Executor: Finished task 0.0 in stage 195.0 (TID 288). 2763 bytes result sent to driver
23/09/24 21:18:30 INFO CoarseGrainedExecutorBackend: Got assigned task 290
23/09/24 21:18:30 INFO CoarseGrainedExecutorBackend: Got assigned task 293
23/09/24 21:18:30 INFO Executor: Running task 0.0 in stage 197.0 (TID 290)
23/09/24 21:18:30 INFO Executor: Running task 3.0 in stage 197.0 (TID 293)
23/09/24 21:18:30 INFO TorrentBroadcast: Started reading broadcast variable 203 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:30 INFO MemoryStore: Block broadcast_203_piece0 stored as bytes in memory (estimated size 7.1 KiB, free 5.0 GiB)
23/09/24 21:18:30 INFO TorrentBroadcast: Reading broadcast variable 203 took 3 ms
23/09/24 21:18:30 INFO MemoryStore: Block broadcast_203 stored as values in memory (estimated size 15.1 KiB, free 5.0 GiB)
23/09/24 21:18:30 INFO BlockManager: Found block rdd_88_3 locally
23/09/24 21:18:30 INFO BlockManager: Found block rdd_88_0 locally
23/09/24 21:18:30 INFO Executor: Finished task 3.0 in stage 197.0 (TID 293). 3309 bytes result sent to driver
23/09/24 21:18:30 INFO Executor: Finished task 0.0 in stage 197.0 (TID 290). 3230 bytes result sent to driver
23/09/24 21:18:30 INFO CoarseGrainedExecutorBackend: Got assigned task 294
23/09/24 21:18:30 INFO CoarseGrainedExecutorBackend: Got assigned task 297
23/09/24 21:18:30 INFO Executor: Running task 0.0 in stage 200.0 (TID 294)
23/09/24 21:18:30 INFO Executor: Running task 3.0 in stage 200.0 (TID 297)
23/09/24 21:18:30 INFO TorrentBroadcast: Started reading broadcast variable 205 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:30 INFO MemoryStore: Block broadcast_205_piece0 stored as bytes in memory (estimated size 52.0 KiB, free 5.0 GiB)
23/09/24 21:18:31 INFO CoarseGrainedExecutorBackend: Got assigned task 298
23/09/24 21:18:31 INFO TorrentBroadcast: Reading broadcast variable 205 took 4 ms
23/09/24 21:18:31 INFO CoarseGrainedExecutorBackend: Got assigned task 301
23/09/24 21:18:31 INFO Executor: Running task 0.0 in stage 202.0 (TID 298)
23/09/24 21:18:31 INFO MemoryStore: Block broadcast_205 stored as values in memory (estimated size 140.3 KiB, free 5.0 GiB)
23/09/24 21:18:31 INFO Executor: Running task 3.0 in stage 202.0 (TID 301)
23/09/24 21:18:31 INFO TorrentBroadcast: Started reading broadcast variable 206 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:31 INFO MemoryStore: Block broadcast_206_piece0 stored as bytes in memory (estimated size 51.9 KiB, free 5.0 GiB)
23/09/24 21:18:31 INFO TorrentBroadcast: Reading broadcast variable 206 took 3 ms
23/09/24 21:18:31 INFO MemoryStore: Block broadcast_206 stored as values in memory (estimated size 140.2 KiB, free 5.0 GiB)
23/09/24 21:18:31 INFO CodeGenerator: Code generated in 12.915056 ms
23/09/24 21:18:31 INFO CodeGenerator: Code generated in 17.533169 ms
23/09/24 21:18:31 INFO BlockManager: Found block rdd_627_0 locally
23/09/24 21:18:31 INFO BlockManager: Found block rdd_627_3 locally
23/09/24 21:18:31 INFO Executor: Finished task 0.0 in stage 200.0 (TID 294). 3001 bytes result sent to driver
23/09/24 21:18:31 INFO BlockManager: Found block rdd_627_0 locally
23/09/24 21:18:31 INFO BlockManager: Found block rdd_627_3 locally
23/09/24 21:18:31 INFO Executor: Finished task 0.0 in stage 202.0 (TID 298). 3125 bytes result sent to driver
23/09/24 21:18:31 INFO Executor: Finished task 3.0 in stage 200.0 (TID 297). 3130 bytes result sent to driver
23/09/24 21:18:31 INFO Executor: Finished task 3.0 in stage 202.0 (TID 301). 3125 bytes result sent to driver
23/09/24 21:18:31 INFO CoarseGrainedExecutorBackend: Got assigned task 304
23/09/24 21:18:31 INFO CoarseGrainedExecutorBackend: Got assigned task 307
23/09/24 21:18:31 INFO CoarseGrainedExecutorBackend: Got assigned task 310
23/09/24 21:18:31 INFO Executor: Running task 2.0 in stage 203.0 (TID 304)
23/09/24 21:18:31 INFO Executor: Running task 5.0 in stage 203.0 (TID 307)
23/09/24 21:18:31 INFO MapOutputTrackerWorker: Updating epoch to 45 and clearing cache
23/09/24 21:18:31 INFO Executor: Running task 8.0 in stage 203.0 (TID 310)
23/09/24 21:18:31 INFO TorrentBroadcast: Started reading broadcast variable 207 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:31 INFO MemoryStore: Block broadcast_207_piece0 stored as bytes in memory (estimated size 53.6 KiB, free 5.0 GiB)
23/09/24 21:18:31 INFO CoarseGrainedExecutorBackend: Got assigned task 315
23/09/24 21:18:31 INFO CoarseGrainedExecutorBackend: Got assigned task 316
23/09/24 21:18:31 INFO Executor: Running task 2.0 in stage 201.0 (TID 315)
23/09/24 21:18:31 INFO MapOutputTrackerWorker: Updating epoch to 46 and clearing cache
23/09/24 21:18:31 INFO TorrentBroadcast: Reading broadcast variable 207 took 4 ms
23/09/24 21:18:31 INFO Executor: Running task 3.0 in stage 201.0 (TID 316)
23/09/24 21:18:31 INFO TorrentBroadcast: Started reading broadcast variable 208 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:31 INFO MemoryStore: Block broadcast_207 stored as values in memory (estimated size 147.5 KiB, free 5.0 GiB)
23/09/24 21:18:31 INFO MemoryStore: Block broadcast_208_piece0 stored as bytes in memory (estimated size 53.6 KiB, free 5.0 GiB)
23/09/24 21:18:31 INFO TorrentBroadcast: Reading broadcast variable 208 took 3 ms
23/09/24 21:18:31 INFO MemoryStore: Block broadcast_208 stored as values in memory (estimated size 147.6 KiB, free 5.0 GiB)
23/09/24 21:18:31 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 44, fetching them
23/09/24 21:18:31 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:41787)
23/09/24 21:18:31 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 46, fetching them
23/09/24 21:18:31 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:41787)
23/09/24 21:18:31 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 44, fetching them
23/09/24 21:18:31 INFO MapOutputTrackerWorker: Got the output locations
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Getting 1 (368.0 B) non-empty blocks including 1 (368.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Getting 1 (405.0 B) non-empty blocks including 0 (0.0 B) local and 1 (405.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Getting 1 (334.0 B) non-empty blocks including 0 (0.0 B) local and 1 (334.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 21:18:31 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 46, fetching them
23/09/24 21:18:31 INFO MapOutputTrackerWorker: Got the output locations
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Getting 1 (334.0 B) non-empty blocks including 1 (334.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Getting 1 (368.0 B) non-empty blocks including 1 (368.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
23/09/24 21:18:31 INFO CodeGenerator: Code generated in 21.170961 ms
23/09/24 21:18:31 INFO CodeGenerator: Code generated in 9.016211 ms
23/09/24 21:18:31 INFO CodeGenerator: Code generated in 10.298258 ms
23/09/24 21:18:31 INFO CodeGenerator: Code generated in 6.889777 ms
23/09/24 21:18:31 INFO MemoryStore: Block rdd_658_2 stored as values in memory (estimated size 4.7 KiB, free 5.0 GiB)
23/09/24 21:18:31 INFO MemoryStore: Block rdd_639_8 stored as values in memory (estimated size 4.8 KiB, free 5.0 GiB)
23/09/24 21:18:31 INFO MemoryStore: Block rdd_639_5 stored as values in memory (estimated size 4.8 KiB, free 5.0 GiB)
23/09/24 21:18:31 INFO MemoryStore: Block rdd_658_3 stored as values in memory (estimated size 4.7 KiB, free 5.0 GiB)
23/09/24 21:18:31 INFO MemoryStore: Block rdd_639_2 stored as values in memory (estimated size 4.7 KiB, free 5.0 GiB)
23/09/24 21:18:31 INFO CodeGenerator: Code generated in 25.478859 ms
23/09/24 21:18:31 INFO CodeGenerator: Code generated in 38.771949 ms
23/09/24 21:18:31 INFO Executor: Finished task 2.0 in stage 203.0 (TID 304). 4580 bytes result sent to driver
23/09/24 21:18:31 INFO Executor: Finished task 8.0 in stage 203.0 (TID 310). 4580 bytes result sent to driver
23/09/24 21:18:31 INFO Executor: Finished task 5.0 in stage 203.0 (TID 307). 4580 bytes result sent to driver
23/09/24 21:18:31 INFO CoarseGrainedExecutorBackend: Got assigned task 322
23/09/24 21:18:31 INFO Executor: Running task 9.0 in stage 201.0 (TID 322)
23/09/24 21:18:31 INFO CoarseGrainedExecutorBackend: Got assigned task 323
23/09/24 21:18:31 INFO CoarseGrainedExecutorBackend: Got assigned task 324
23/09/24 21:18:31 INFO Executor: Finished task 2.0 in stage 201.0 (TID 315). 4580 bytes result sent to driver
23/09/24 21:18:31 INFO Executor: Running task 11.0 in stage 201.0 (TID 324)
23/09/24 21:18:31 INFO Executor: Running task 10.0 in stage 201.0 (TID 323)
23/09/24 21:18:31 INFO CoarseGrainedExecutorBackend: Got assigned task 325
23/09/24 21:18:31 INFO Executor: Finished task 3.0 in stage 201.0 (TID 316). 4580 bytes result sent to driver
23/09/24 21:18:31 INFO Executor: Running task 12.0 in stage 201.0 (TID 325)
23/09/24 21:18:31 INFO CoarseGrainedExecutorBackend: Got assigned task 326
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Getting 1 (445.0 B) non-empty blocks including 0 (0.0 B) local and 1 (445.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 21:18:31 INFO MemoryStore: Block rdd_658_11 stored as values in memory (estimated size 4.8 KiB, free 5.0 GiB)
23/09/24 21:18:31 INFO Executor: Running task 13.0 in stage 201.0 (TID 326)
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Getting 1 (445.0 B) non-empty blocks including 0 (0.0 B) local and 1 (445.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
23/09/24 21:18:31 INFO Executor: Finished task 11.0 in stage 201.0 (TID 324). 4537 bytes result sent to driver
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Getting 1 (445.0 B) non-empty blocks including 0 (0.0 B) local and 1 (445.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 21:18:31 INFO MemoryStore: Block rdd_658_10 stored as values in memory (estimated size 4.8 KiB, free 4.9 GiB)
23/09/24 21:18:31 INFO CoarseGrainedExecutorBackend: Got assigned task 327
23/09/24 21:18:31 INFO Executor: Running task 14.0 in stage 201.0 (TID 327)
23/09/24 21:18:31 INFO MemoryStore: Block rdd_658_9 stored as values in memory (estimated size 4.8 KiB, free 5.0 GiB)
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Getting 1 (445.0 B) non-empty blocks including 1 (445.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
23/09/24 21:18:31 INFO MemoryStore: Block rdd_658_12 stored as values in memory (estimated size 4.8 KiB, free 5.0 GiB)
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Getting 1 (445.0 B) non-empty blocks including 1 (445.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 21:18:31 INFO Executor: Finished task 9.0 in stage 201.0 (TID 322). 4537 bytes result sent to driver
23/09/24 21:18:31 INFO Executor: Finished task 10.0 in stage 201.0 (TID 323). 4537 bytes result sent to driver
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Getting 1 (445.0 B) non-empty blocks including 1 (445.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 21:18:31 INFO Executor: Finished task 12.0 in stage 201.0 (TID 325). 4537 bytes result sent to driver
23/09/24 21:18:31 INFO MemoryStore: Block rdd_658_14 stored as values in memory (estimated size 4.8 KiB, free 5.0 GiB)
23/09/24 21:18:31 INFO MemoryStore: Block rdd_658_13 stored as values in memory (estimated size 4.8 KiB, free 5.0 GiB)
23/09/24 21:18:31 INFO Executor: Finished task 14.0 in stage 201.0 (TID 327). 4537 bytes result sent to driver
23/09/24 21:18:31 INFO Executor: Finished task 13.0 in stage 201.0 (TID 326). 4537 bytes result sent to driver
23/09/24 21:18:31 INFO CoarseGrainedExecutorBackend: Got assigned task 329
23/09/24 21:18:31 INFO CoarseGrainedExecutorBackend: Got assigned task 332
23/09/24 21:18:31 INFO Executor: Running task 0.0 in stage 204.0 (TID 329)
23/09/24 21:18:31 INFO Executor: Running task 3.0 in stage 204.0 (TID 332)
23/09/24 21:18:31 INFO MapOutputTrackerWorker: Updating epoch to 48 and clearing cache
23/09/24 21:18:31 INFO TorrentBroadcast: Started reading broadcast variable 209 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:31 INFO MemoryStore: Block broadcast_209_piece0 stored as bytes in memory (estimated size 75.1 KiB, free 5.0 GiB)
23/09/24 21:18:31 INFO TorrentBroadcast: Reading broadcast variable 209 took 3 ms
23/09/24 21:18:31 INFO MemoryStore: Block broadcast_209 stored as values in memory (estimated size 233.4 KiB, free 5.0 GiB)
23/09/24 21:18:31 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 45, fetching them
23/09/24 21:18:31 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:41787)
23/09/24 21:18:31 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 47, fetching them
23/09/24 21:18:31 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:41787)
23/09/24 21:18:31 INFO MapOutputTrackerWorker: Got the output locations
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Getting 11 (4.0 KiB) non-empty blocks including 3 (1107.0 B) local and 8 (2.9 KiB) host-local and 0 (0.0 B) remote blocks
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 21:18:31 INFO MapOutputTrackerWorker: Got the output locations
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Getting 16 (6.5 KiB) non-empty blocks including 8 (3.3 KiB) local and 8 (3.2 KiB) host-local and 0 (0.0 B) remote blocks
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 21:18:31 INFO CodeGenerator: Code generated in 6.566149 ms
23/09/24 21:18:31 INFO CodeGenerator: Code generated in 6.156054 ms
23/09/24 21:18:31 INFO CodeGenerator: Code generated in 8.770667 ms
23/09/24 21:18:31 INFO CodeGenerator: Code generated in 8.013138 ms
23/09/24 21:18:31 INFO CodeGenerator: Code generated in 29.023336 ms
23/09/24 21:18:31 INFO MemoryStore: Block rdd_679_1 stored as values in memory (estimated size 7.5 KiB, free 5.0 GiB)
23/09/24 21:18:31 INFO MemoryStore: Block rdd_679_0 stored as values in memory (estimated size 7.6 KiB, free 5.0 GiB)
23/09/24 21:18:31 INFO CodeGenerator: Code generated in 10.25153 ms
23/09/24 21:18:31 INFO CodeGenerator: Code generated in 11.019559 ms
23/09/24 21:18:31 INFO CodeGenerator: Code generated in 4.535939 ms
23/09/24 21:18:31 INFO MemoryStore: Block rdd_689_1 stored as values in memory (estimated size 4.2 KiB, free 5.0 GiB)
23/09/24 21:18:31 INFO CodeGenerator: Code generated in 15.750626 ms
23/09/24 21:18:31 INFO CodeGenerator: Code generated in 7.202889 ms
23/09/24 21:18:31 INFO CodeGenerator: Code generated in 7.005226 ms
23/09/24 21:18:31 INFO TorrentBroadcast: Started reading broadcast variable 204 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:31 INFO MemoryStore: Block broadcast_204_piece0 stored as bytes in memory (estimated size 3.7 KiB, free 5.0 GiB)
23/09/24 21:18:31 INFO TorrentBroadcast: Reading broadcast variable 204 took 3 ms
23/09/24 21:18:31 INFO MemoryStore: Block broadcast_204 stored as values in memory (estimated size 7.6 KiB, free 5.0 GiB)
23/09/24 21:18:31 INFO CodeGenerator: Code generated in 4.969216 ms
23/09/24 21:18:31 INFO Executor: Finished task 0.0 in stage 204.0 (TID 329). 7923 bytes result sent to driver
23/09/24 21:18:31 INFO Executor: Finished task 3.0 in stage 204.0 (TID 332). 7923 bytes result sent to driver
23/09/24 21:18:31 INFO CoarseGrainedExecutorBackend: Got assigned task 333
23/09/24 21:18:31 INFO CoarseGrainedExecutorBackend: Got assigned task 335
23/09/24 21:18:31 INFO Executor: Running task 0.0 in stage 211.0 (TID 333)
23/09/24 21:18:31 INFO CoarseGrainedExecutorBackend: Got assigned task 336
23/09/24 21:18:31 INFO Executor: Running task 1.0 in stage 211.0 (TID 335)
23/09/24 21:18:31 INFO Executor: Running task 3.0 in stage 211.0 (TID 336)
23/09/24 21:18:31 INFO TorrentBroadcast: Started reading broadcast variable 210 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:31 INFO MemoryStore: Block broadcast_210_piece0 stored as bytes in memory (estimated size 75.3 KiB, free 5.0 GiB)
23/09/24 21:18:31 INFO TorrentBroadcast: Reading broadcast variable 210 took 2 ms
23/09/24 21:18:31 INFO MemoryStore: Block broadcast_210 stored as values in memory (estimated size 234.1 KiB, free 5.0 GiB)
23/09/24 21:18:31 INFO CodeGenerator: Code generated in 4.399791 ms
23/09/24 21:18:31 INFO BlockManager: Found block rdd_679_0 locally
23/09/24 21:18:31 INFO BlockManager: Found block rdd_679_1 locally
23/09/24 21:18:31 INFO BlockManager: Found block rdd_689_1 locally
23/09/24 21:18:31 INFO Executor: Finished task 1.0 in stage 211.0 (TID 335). 7948 bytes result sent to driver
23/09/24 21:18:31 INFO Executor: Finished task 3.0 in stage 211.0 (TID 336). 7948 bytes result sent to driver
23/09/24 21:18:31 INFO Executor: Finished task 0.0 in stage 211.0 (TID 333). 7948 bytes result sent to driver
23/09/24 21:18:31 INFO CoarseGrainedExecutorBackend: Got assigned task 338
23/09/24 21:18:31 INFO CoarseGrainedExecutorBackend: Got assigned task 341
23/09/24 21:18:31 INFO Executor: Running task 1.0 in stage 212.0 (TID 338)
23/09/24 21:18:31 INFO Executor: Running task 4.0 in stage 212.0 (TID 341)
23/09/24 21:18:31 INFO CoarseGrainedExecutorBackend: Got assigned task 344
23/09/24 21:18:31 INFO CoarseGrainedExecutorBackend: Got assigned task 347
23/09/24 21:18:31 INFO CoarseGrainedExecutorBackend: Got assigned task 350
23/09/24 21:18:31 INFO Executor: Running task 10.0 in stage 212.0 (TID 347)
23/09/24 21:18:31 INFO MapOutputTrackerWorker: Updating epoch to 49 and clearing cache
23/09/24 21:18:31 INFO Executor: Running task 7.0 in stage 212.0 (TID 344)
23/09/24 21:18:31 INFO Executor: Running task 13.0 in stage 212.0 (TID 350)
23/09/24 21:18:31 INFO TorrentBroadcast: Started reading broadcast variable 211 with 1 pieces (estimated total size 4.0 MiB)
23/09/24 21:18:31 INFO MemoryStore: Block broadcast_211_piece0 stored as bytes in memory (estimated size 129.5 KiB, free 5.0 GiB)
23/09/24 21:18:31 INFO TorrentBroadcast: Reading broadcast variable 211 took 4 ms
23/09/24 21:18:31 INFO MemoryStore: Block broadcast_211 stored as values in memory (estimated size 379.7 KiB, free 5.0 GiB)
23/09/24 21:18:31 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 48, fetching them
23/09/24 21:18:31 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:41787)
23/09/24 21:18:31 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 48, fetching them
23/09/24 21:18:31 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 48, fetching them
23/09/24 21:18:31 INFO MapOutputTrackerWorker: Got the output locations
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Getting 2 (552.0 B) non-empty blocks including 2 (552.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Getting 2 (608.0 B) non-empty blocks including 1 (304.0 B) local and 1 (304.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Getting 2 (552.0 B) non-empty blocks including 2 (552.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Getting 2 (608.0 B) non-empty blocks including 1 (304.0 B) local and 1 (304.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Getting 2 (580.0 B) non-empty blocks including 2 (580.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 21:18:31 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
23/09/24 21:18:31 INFO CodeGenerator: Code generated in 16.615627 ms
23/09/24 21:18:31 INFO CodeGenerator: Code generated in 8.94155 ms
23/09/24 21:18:31 INFO CodeGenerator: Code generated in 9.213212 ms
23/09/24 21:18:31 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 21:18:31 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 21:18:31 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 21:18:31 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 21:18:31 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 21:18:31 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 21:18:31 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 21:18:31 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 21:18:31 INFO CodecConfig: Compression: SNAPPY
23/09/24 21:18:31 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 21:18:31 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 21:18:31 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 21:18:31 INFO CodecConfig: Compression: SNAPPY
23/09/24 21:18:31 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 21:18:31 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 21:18:31 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 21:18:31 INFO CodecConfig: Compression: SNAPPY
23/09/24 21:18:31 INFO CodecConfig: Compression: SNAPPY
23/09/24 21:18:31 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 21:18:31 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 21:18:31 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 21:18:31 INFO CodecConfig: Compression: SNAPPY
23/09/24 21:18:31 INFO CodecConfig: Compression: SNAPPY
23/09/24 21:18:31 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 21:18:31 INFO CodecConfig: Compression: SNAPPY
23/09/24 21:18:31 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 21:18:31 INFO CodecConfig: Compression: SNAPPY
23/09/24 21:18:31 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 21:18:31 INFO CodecConfig: Compression: SNAPPY
23/09/24 21:18:31 INFO CodecConfig: Compression: SNAPPY
23/09/24 21:18:32 INFO ParquetOutputFormat: Parquet block size to 134217728
23/09/24 21:18:32 INFO ParquetOutputFormat: Parquet page size to 1048576
23/09/24 21:18:32 INFO ParquetOutputFormat: Parquet dictionary page size to 1048576
23/09/24 21:18:32 INFO ParquetOutputFormat: Dictionary is on
23/09/24 21:18:32 INFO ParquetOutputFormat: Validation is off
23/09/24 21:18:32 INFO ParquetOutputFormat: Writer version is: PARQUET_1_0
23/09/24 21:18:32 INFO ParquetOutputFormat: Maximum row group padding size is 8388608 bytes
23/09/24 21:18:32 INFO ParquetOutputFormat: Page size checking is: estimated
23/09/24 21:18:32 INFO ParquetOutputFormat: Min row count for page size check is: 100
23/09/24 21:18:32 INFO ParquetOutputFormat: Max row count for page size check is: 10000
23/09/24 21:18:32 INFO ParquetOutputFormat: Parquet block size to 134217728
23/09/24 21:18:32 INFO ParquetOutputFormat: Parquet page size to 1048576
23/09/24 21:18:32 INFO ParquetOutputFormat: Parquet dictionary page size to 1048576
23/09/24 21:18:32 INFO ParquetOutputFormat: Dictionary is on
23/09/24 21:18:32 INFO ParquetOutputFormat: Validation is off
23/09/24 21:18:32 INFO ParquetOutputFormat: Writer version is: PARQUET_1_0
23/09/24 21:18:32 INFO ParquetOutputFormat: Maximum row group padding size is 8388608 bytes
23/09/24 21:18:32 INFO ParquetOutputFormat: Page size checking is: estimated
23/09/24 21:18:32 INFO ParquetOutputFormat: Min row count for page size check is: 100
23/09/24 21:18:32 INFO ParquetOutputFormat: Max row count for page size check is: 10000
23/09/24 21:18:32 INFO ParquetOutputFormat: Parquet block size to 134217728
23/09/24 21:18:32 INFO ParquetOutputFormat: Parquet page size to 1048576
23/09/24 21:18:32 INFO ParquetOutputFormat: Parquet dictionary page size to 1048576
23/09/24 21:18:32 INFO ParquetOutputFormat: Dictionary is on
23/09/24 21:18:32 INFO ParquetOutputFormat: Validation is off
23/09/24 21:18:32 INFO ParquetOutputFormat: Writer version is: PARQUET_1_0
23/09/24 21:18:32 INFO ParquetOutputFormat: Maximum row group padding size is 8388608 bytes
23/09/24 21:18:32 INFO ParquetOutputFormat: Page size checking is: estimated
23/09/24 21:18:32 INFO ParquetOutputFormat: Min row count for page size check is: 100
23/09/24 21:18:32 INFO ParquetOutputFormat: Max row count for page size check is: 10000
23/09/24 21:18:32 INFO ParquetOutputFormat: Parquet block size to 134217728
23/09/24 21:18:32 INFO ParquetOutputFormat: Parquet page size to 1048576
23/09/24 21:18:32 INFO ParquetOutputFormat: Parquet dictionary page size to 1048576
23/09/24 21:18:32 INFO ParquetOutputFormat: Dictionary is on
23/09/24 21:18:32 INFO ParquetOutputFormat: Validation is off
23/09/24 21:18:32 INFO ParquetOutputFormat: Writer version is: PARQUET_1_0
23/09/24 21:18:32 INFO ParquetOutputFormat: Maximum row group padding size is 8388608 bytes
23/09/24 21:18:32 INFO ParquetOutputFormat: Page size checking is: estimated
23/09/24 21:18:32 INFO ParquetOutputFormat: Min row count for page size check is: 100
23/09/24 21:18:32 INFO ParquetOutputFormat: Max row count for page size check is: 10000
23/09/24 21:18:32 INFO ParquetOutputFormat: Parquet block size to 134217728
23/09/24 21:18:32 INFO ParquetOutputFormat: Parquet page size to 1048576
23/09/24 21:18:32 INFO ParquetOutputFormat: Parquet dictionary page size to 1048576
23/09/24 21:18:32 INFO ParquetOutputFormat: Dictionary is on
23/09/24 21:18:32 INFO ParquetOutputFormat: Validation is off
23/09/24 21:18:32 INFO ParquetOutputFormat: Writer version is: PARQUET_1_0
23/09/24 21:18:32 INFO ParquetOutputFormat: Maximum row group padding size is 8388608 bytes
23/09/24 21:18:32 INFO ParquetOutputFormat: Page size checking is: estimated
23/09/24 21:18:32 INFO ParquetOutputFormat: Min row count for page size check is: 100
23/09/24 21:18:32 INFO ParquetOutputFormat: Max row count for page size check is: 10000
23/09/24 21:18:32 INFO ParquetWriteSupport: Initialized Parquet WriteSupport with Catalyst schema:
{
  "type" : "struct",
  "fields" : [ {
    "name" : "z_zid",
    "type" : "long",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "z_cluster",
    "type" : "string",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "z_prediction",
    "type" : "double",
    "nullable" : false,
    "metadata" : {
      "ml_attr" : {
        "type" : "nominal",
        "num_vals" : 2
      }
    }
  }, {
    "name" : "z_score",
    "type" : "double",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "z_isMatch",
    "type" : "integer",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "fname",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "lname",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "stNo",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "add1",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "add2",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "city",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "areacode",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "state",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "dob",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "ssn",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "z_source",
    "type" : "string",
    "nullable" : false,
    "metadata" : { }
  } ]
}
and corresponding Parquet message type:
message spark_schema {
  required int64 z_zid;
  required binary z_cluster (UTF8);
  required double z_prediction;
  optional double z_score;
  required int32 z_isMatch;
  optional binary fname (UTF8);
  optional binary lname (UTF8);
  optional binary stNo (UTF8);
  optional binary add1 (UTF8);
  optional binary add2 (UTF8);
  optional binary city (UTF8);
  optional binary areacode (UTF8);
  optional binary state (UTF8);
  optional binary dob (UTF8);
  optional binary ssn (UTF8);
  required binary z_source (UTF8);
}

       
23/09/24 21:18:32 INFO ParquetWriteSupport: Initialized Parquet WriteSupport with Catalyst schema:
{
  "type" : "struct",
  "fields" : [ {
    "name" : "z_zid",
    "type" : "long",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "z_cluster",
    "type" : "string",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "z_prediction",
    "type" : "double",
    "nullable" : false,
    "metadata" : {
      "ml_attr" : {
        "type" : "nominal",
        "num_vals" : 2
      }
    }
  }, {
    "name" : "z_score",
    "type" : "double",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "z_isMatch",
    "type" : "integer",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "fname",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "lname",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "stNo",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "add1",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "add2",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "city",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "areacode",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "state",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "dob",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "ssn",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "z_source",
    "type" : "string",
    "nullable" : false,
    "metadata" : { }
  } ]
}
and corresponding Parquet message type:
message spark_schema {
  required int64 z_zid;
  required binary z_cluster (UTF8);
  required double z_prediction;
  optional double z_score;
  required int32 z_isMatch;
  optional binary fname (UTF8);
  optional binary lname (UTF8);
  optional binary stNo (UTF8);
  optional binary add1 (UTF8);
  optional binary add2 (UTF8);
  optional binary city (UTF8);
  optional binary areacode (UTF8);
  optional binary state (UTF8);
  optional binary dob (UTF8);
  optional binary ssn (UTF8);
  required binary z_source (UTF8);
}

       
23/09/24 21:18:32 INFO ParquetWriteSupport: Initialized Parquet WriteSupport with Catalyst schema:
{
  "type" : "struct",
  "fields" : [ {
    "name" : "z_zid",
    "type" : "long",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "z_cluster",
    "type" : "string",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "z_prediction",
    "type" : "double",
    "nullable" : false,
    "metadata" : {
      "ml_attr" : {
        "type" : "nominal",
        "num_vals" : 2
      }
    }
  }, {
    "name" : "z_score",
    "type" : "double",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "z_isMatch",
    "type" : "integer",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "fname",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "lname",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "stNo",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "add1",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "add2",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "city",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "areacode",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "state",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "dob",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "ssn",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "z_source",
    "type" : "string",
    "nullable" : false,
    "metadata" : { }
  } ]
}
and corresponding Parquet message type:
message spark_schema {
  required int64 z_zid;
  required binary z_cluster (UTF8);
  required double z_prediction;
  optional double z_score;
  required int32 z_isMatch;
  optional binary fname (UTF8);
  optional binary lname (UTF8);
  optional binary stNo (UTF8);
  optional binary add1 (UTF8);
  optional binary add2 (UTF8);
  optional binary city (UTF8);
  optional binary areacode (UTF8);
  optional binary state (UTF8);
  optional binary dob (UTF8);
  optional binary ssn (UTF8);
  required binary z_source (UTF8);
}

       
23/09/24 21:18:32 INFO ParquetWriteSupport: Initialized Parquet WriteSupport with Catalyst schema:
{
  "type" : "struct",
  "fields" : [ {
    "name" : "z_zid",
    "type" : "long",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "z_cluster",
    "type" : "string",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "z_prediction",
    "type" : "double",
    "nullable" : false,
    "metadata" : {
      "ml_attr" : {
        "type" : "nominal",
        "num_vals" : 2
      }
    }
  }, {
    "name" : "z_score",
    "type" : "double",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "z_isMatch",
    "type" : "integer",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "fname",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "lname",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "stNo",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "add1",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "add2",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "city",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "areacode",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "state",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "dob",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "ssn",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "z_source",
    "type" : "string",
    "nullable" : false,
    "metadata" : { }
  } ]
}
and corresponding Parquet message type:
message spark_schema {
  required int64 z_zid;
  required binary z_cluster (UTF8);
  required double z_prediction;
  optional double z_score;
  required int32 z_isMatch;
  optional binary fname (UTF8);
  optional binary lname (UTF8);
  optional binary stNo (UTF8);
  optional binary add1 (UTF8);
  optional binary add2 (UTF8);
  optional binary city (UTF8);
  optional binary areacode (UTF8);
  optional binary state (UTF8);
  optional binary dob (UTF8);
  optional binary ssn (UTF8);
  required binary z_source (UTF8);
}

       
23/09/24 21:18:32 INFO ParquetWriteSupport: Initialized Parquet WriteSupport with Catalyst schema:
{
  "type" : "struct",
  "fields" : [ {
    "name" : "z_zid",
    "type" : "long",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "z_cluster",
    "type" : "string",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "z_prediction",
    "type" : "double",
    "nullable" : false,
    "metadata" : {
      "ml_attr" : {
        "type" : "nominal",
        "num_vals" : 2
      }
    }
  }, {
    "name" : "z_score",
    "type" : "double",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "z_isMatch",
    "type" : "integer",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "fname",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "lname",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "stNo",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "add1",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "add2",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "city",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "areacode",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "state",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "dob",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "ssn",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "z_source",
    "type" : "string",
    "nullable" : false,
    "metadata" : { }
  } ]
}
and corresponding Parquet message type:
message spark_schema {
  required int64 z_zid;
  required binary z_cluster (UTF8);
  required double z_prediction;
  optional double z_score;
  required int32 z_isMatch;
  optional binary fname (UTF8);
  optional binary lname (UTF8);
  optional binary stNo (UTF8);
  optional binary add1 (UTF8);
  optional binary add2 (UTF8);
  optional binary city (UTF8);
  optional binary areacode (UTF8);
  optional binary state (UTF8);
  optional binary dob (UTF8);
  optional binary ssn (UTF8);
  required binary z_source (UTF8);
}

       
23/09/24 21:18:32 INFO CodecPool: Got brand-new compressor [.snappy]
23/09/24 21:18:32 INFO CodecPool: Got brand-new compressor [.snappy]
23/09/24 21:18:32 INFO CodecPool: Got brand-new compressor [.snappy]
23/09/24 21:18:32 INFO CodecPool: Got brand-new compressor [.snappy]
23/09/24 21:18:32 INFO CodecPool: Got brand-new compressor [.snappy]
23/09/24 21:18:32 INFO InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 421
23/09/24 21:18:32 INFO InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 331
23/09/24 21:18:32 INFO InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 313
23/09/24 21:18:32 INFO InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 359
23/09/24 21:18:32 INFO InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 310
23/09/24 21:18:32 INFO FileOutputCommitter: Saved output of task 'attempt_202309242118315590538880695593611_0212_m_000001_338' to file:/root/zingg/models/100/trainingData/unmarked/_temporary/0/task_202309242118315590538880695593611_0212_m_000001
23/09/24 21:18:32 INFO FileOutputCommitter: Saved output of task 'attempt_202309242118318489915263937009529_0212_m_000007_344' to file:/root/zingg/models/100/trainingData/unmarked/_temporary/0/task_202309242118318489915263937009529_0212_m_000007
23/09/24 21:18:32 INFO FileOutputCommitter: Saved output of task 'attempt_202309242118317721427837990338013_0212_m_000013_350' to file:/root/zingg/models/100/trainingData/unmarked/_temporary/0/task_202309242118317721427837990338013_0212_m_000013
23/09/24 21:18:32 INFO SparkHadoopMapRedUtil: attempt_202309242118315590538880695593611_0212_m_000001_338: Committed
23/09/24 21:18:32 INFO SparkHadoopMapRedUtil: attempt_202309242118318489915263937009529_0212_m_000007_344: Committed
23/09/24 21:18:32 INFO FileOutputCommitter: Saved output of task 'attempt_202309242118311268394132395903605_0212_m_000004_341' to file:/root/zingg/models/100/trainingData/unmarked/_temporary/0/task_202309242118311268394132395903605_0212_m_000004
23/09/24 21:18:32 INFO SparkHadoopMapRedUtil: attempt_202309242118311268394132395903605_0212_m_000004_341: Committed
23/09/24 21:18:32 INFO SparkHadoopMapRedUtil: attempt_202309242118317721427837990338013_0212_m_000013_350: Committed
23/09/24 21:18:32 INFO Executor: Finished task 1.0 in stage 212.0 (TID 338). 10148 bytes result sent to driver
23/09/24 21:18:32 INFO FileOutputCommitter: Saved output of task 'attempt_202309242118314826568095541239054_0212_m_000010_347' to file:/root/zingg/models/100/trainingData/unmarked/_temporary/0/task_202309242118314826568095541239054_0212_m_000010
23/09/24 21:18:32 INFO SparkHadoopMapRedUtil: attempt_202309242118314826568095541239054_0212_m_000010_347: Committed
23/09/24 21:18:32 INFO Executor: Finished task 10.0 in stage 212.0 (TID 347). 10148 bytes result sent to driver
23/09/24 21:18:32 INFO Executor: Finished task 7.0 in stage 212.0 (TID 344). 10148 bytes result sent to driver
23/09/24 21:18:32 INFO Executor: Finished task 4.0 in stage 212.0 (TID 341). 10148 bytes result sent to driver
23/09/24 21:18:32 INFO Executor: Finished task 13.0 in stage 212.0 (TID 350). 10148 bytes result sent to driver
23/09/24 21:18:32 INFO CoarseGrainedExecutorBackend: Got assigned task 352
23/09/24 21:18:32 INFO CoarseGrainedExecutorBackend: Got assigned task 353
23/09/24 21:18:32 INFO CoarseGrainedExecutorBackend: Got assigned task 354
23/09/24 21:18:32 INFO Executor: Running task 16.0 in stage 212.0 (TID 353)
23/09/24 21:18:32 INFO CoarseGrainedExecutorBackend: Got assigned task 355
23/09/24 21:18:32 INFO Executor: Running task 17.0 in stage 212.0 (TID 354)
23/09/24 21:18:32 INFO CoarseGrainedExecutorBackend: Got assigned task 356
23/09/24 21:18:32 INFO Executor: Running task 18.0 in stage 212.0 (TID 355)
23/09/24 21:18:32 INFO Executor: Running task 15.0 in stage 212.0 (TID 352)
23/09/24 21:18:32 INFO Executor: Running task 19.0 in stage 212.0 (TID 356)
23/09/24 21:18:32 INFO ShuffleBlockFetcherIterator: Getting 2 (608.0 B) non-empty blocks including 1 (304.0 B) local and 1 (304.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 21:18:32 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 21:18:32 INFO ShuffleBlockFetcherIterator: Getting 2 (608.0 B) non-empty blocks including 2 (608.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 21:18:32 INFO ShuffleBlockFetcherIterator: Getting 2 (608.0 B) non-empty blocks including 2 (608.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 21:18:32 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 21:18:32 INFO ShuffleBlockFetcherIterator: Getting 2 (580.0 B) non-empty blocks including 2 (580.0 B) local and 0 (0.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 21:18:32 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 21:18:32 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms
23/09/24 21:18:32 INFO ShuffleBlockFetcherIterator: Getting 2 (580.0 B) non-empty blocks including 1 (304.0 B) local and 1 (276.0 B) host-local and 0 (0.0 B) remote blocks
23/09/24 21:18:32 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms
23/09/24 21:18:32 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 21:18:32 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 21:18:32 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 21:18:32 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 21:18:32 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 21:18:32 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 21:18:32 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 21:18:32 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 21:18:32 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 21:18:32 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 21:18:32 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 21:18:32 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 21:18:32 INFO CodecConfig: Compression: SNAPPY
23/09/24 21:18:32 INFO CodecConfig: Compression: SNAPPY
23/09/24 21:18:32 INFO CodecConfig: Compression: SNAPPY
23/09/24 21:18:32 INFO CodecConfig: Compression: SNAPPY
23/09/24 21:18:32 INFO ParquetOutputFormat: Parquet block size to 134217728
23/09/24 21:18:32 INFO ParquetOutputFormat: Parquet page size to 1048576
23/09/24 21:18:32 INFO ParquetOutputFormat: Parquet dictionary page size to 1048576
23/09/24 21:18:32 INFO CodecConfig: Compression: SNAPPY
23/09/24 21:18:32 INFO ParquetOutputFormat: Dictionary is on
23/09/24 21:18:32 INFO ParquetOutputFormat: Validation is off
23/09/24 21:18:32 INFO CodecConfig: Compression: SNAPPY
23/09/24 21:18:32 INFO ParquetOutputFormat: Writer version is: PARQUET_1_0
23/09/24 21:18:32 INFO ParquetOutputFormat: Maximum row group padding size is 8388608 bytes
23/09/24 21:18:32 INFO ParquetOutputFormat: Parquet block size to 134217728
23/09/24 21:18:32 INFO ParquetOutputFormat: Parquet page size to 1048576
23/09/24 21:18:32 INFO ParquetOutputFormat: Parquet dictionary page size to 1048576
23/09/24 21:18:32 INFO ParquetOutputFormat: Page size checking is: estimated
23/09/24 21:18:32 INFO ParquetOutputFormat: Min row count for page size check is: 100
23/09/24 21:18:32 INFO ParquetOutputFormat: Dictionary is on
23/09/24 21:18:32 INFO ParquetOutputFormat: Parquet block size to 134217728
23/09/24 21:18:32 INFO ParquetOutputFormat: Max row count for page size check is: 10000
23/09/24 21:18:32 INFO ParquetOutputFormat: Parquet page size to 1048576
23/09/24 21:18:32 INFO ParquetOutputFormat: Parquet dictionary page size to 1048576
23/09/24 21:18:32 INFO ParquetOutputFormat: Validation is off
23/09/24 21:18:32 INFO ParquetOutputFormat: Dictionary is on
23/09/24 21:18:32 INFO ParquetOutputFormat: Validation is off
23/09/24 21:18:32 INFO ParquetOutputFormat: Writer version is: PARQUET_1_0
23/09/24 21:18:32 INFO ParquetOutputFormat: Maximum row group padding size is 8388608 bytes
23/09/24 21:18:32 INFO ParquetOutputFormat: Page size checking is: estimated
23/09/24 21:18:32 INFO ParquetOutputFormat: Min row count for page size check is: 100
23/09/24 21:18:32 INFO ParquetOutputFormat: Max row count for page size check is: 10000
23/09/24 21:18:32 INFO ParquetOutputFormat: Writer version is: PARQUET_1_0
23/09/24 21:18:32 INFO ParquetOutputFormat: Maximum row group padding size is 8388608 bytes
23/09/24 21:18:32 INFO ParquetOutputFormat: Page size checking is: estimated
23/09/24 21:18:32 INFO ParquetOutputFormat: Min row count for page size check is: 100
23/09/24 21:18:32 INFO ParquetOutputFormat: Max row count for page size check is: 10000
23/09/24 21:18:32 INFO ParquetWriteSupport: Initialized Parquet WriteSupport with Catalyst schema:
{
  "type" : "struct",
  "fields" : [ {
    "name" : "z_zid",
    "type" : "long",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "z_cluster",
    "type" : "string",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "z_prediction",
    "type" : "double",
    "nullable" : false,
    "metadata" : {
      "ml_attr" : {
        "type" : "nominal",
        "num_vals" : 2
      }
    }
  }, {
    "name" : "z_score",
    "type" : "double",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "z_isMatch",
    "type" : "integer",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "fname",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "lname",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "stNo",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "add1",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "add2",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "city",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "areacode",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "state",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "dob",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "ssn",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "z_source",
    "type" : "string",
    "nullable" : false,
    "metadata" : { }
  } ]
}
and corresponding Parquet message type:
message spark_schema {
  required int64 z_zid;
  required binary z_cluster (UTF8);
  required double z_prediction;
  optional double z_score;
  required int32 z_isMatch;
  optional binary fname (UTF8);
  optional binary lname (UTF8);
  optional binary stNo (UTF8);
  optional binary add1 (UTF8);
  optional binary add2 (UTF8);
  optional binary city (UTF8);
  optional binary areacode (UTF8);
  optional binary state (UTF8);
  optional binary dob (UTF8);
  optional binary ssn (UTF8);
  required binary z_source (UTF8);
}

       
23/09/24 21:18:32 INFO ParquetWriteSupport: Initialized Parquet WriteSupport with Catalyst schema:
{
  "type" : "struct",
  "fields" : [ {
    "name" : "z_zid",
    "type" : "long",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "z_cluster",
    "type" : "string",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "z_prediction",
    "type" : "double",
    "nullable" : false,
    "metadata" : {
      "ml_attr" : {
        "type" : "nominal",
        "num_vals" : 2
      }
    }
  }, {
    "name" : "z_score",
    "type" : "double",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "z_isMatch",
    "type" : "integer",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "fname",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "lname",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "stNo",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "add1",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "add2",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "city",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "areacode",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "state",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "dob",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "ssn",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "z_source",
    "type" : "string",
    "nullable" : false,
    "metadata" : { }
  } ]
}
and corresponding Parquet message type:
message spark_schema {
  required int64 z_zid;
  required binary z_cluster (UTF8);
  required double z_prediction;
  optional double z_score;
  required int32 z_isMatch;
  optional binary fname (UTF8);
  optional binary lname (UTF8);
  optional binary stNo (UTF8);
  optional binary add1 (UTF8);
  optional binary add2 (UTF8);
  optional binary city (UTF8);
  optional binary areacode (UTF8);
  optional binary state (UTF8);
  optional binary dob (UTF8);
  optional binary ssn (UTF8);
  required binary z_source (UTF8);
}

       
23/09/24 21:18:32 INFO ParquetWriteSupport: Initialized Parquet WriteSupport with Catalyst schema:
{
  "type" : "struct",
  "fields" : [ {
    "name" : "z_zid",
    "type" : "long",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "z_cluster",
    "type" : "string",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "z_prediction",
    "type" : "double",
    "nullable" : false,
    "metadata" : {
      "ml_attr" : {
        "type" : "nominal",
        "num_vals" : 2
      }
    }
  }, {
    "name" : "z_score",
    "type" : "double",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "z_isMatch",
    "type" : "integer",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "fname",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "lname",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "stNo",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "add1",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "add2",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "city",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "areacode",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "state",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "dob",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "ssn",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "z_source",
    "type" : "string",
    "nullable" : false,
    "metadata" : { }
  } ]
}
and corresponding Parquet message type:
message spark_schema {
  required int64 z_zid;
  required binary z_cluster (UTF8);
  required double z_prediction;
  optional double z_score;
  required int32 z_isMatch;
  optional binary fname (UTF8);
  optional binary lname (UTF8);
  optional binary stNo (UTF8);
  optional binary add1 (UTF8);
  optional binary add2 (UTF8);
  optional binary city (UTF8);
  optional binary areacode (UTF8);
  optional binary state (UTF8);
  optional binary dob (UTF8);
  optional binary ssn (UTF8);
  required binary z_source (UTF8);
}

       
23/09/24 21:18:32 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 21:18:32 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 21:18:32 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 21:18:32 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 21:18:32 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 21:18:32 INFO SQLHadoopMapReduceCommitProtocol: Using user defined output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 21:18:32 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
23/09/24 21:18:32 INFO SQLHadoopMapReduceCommitProtocol: Using output committer class org.apache.parquet.hadoop.ParquetOutputCommitter
23/09/24 21:18:32 INFO CodecConfig: Compression: SNAPPY
23/09/24 21:18:32 INFO CodecConfig: Compression: SNAPPY
23/09/24 21:18:32 INFO ParquetOutputFormat: Parquet block size to 134217728
23/09/24 21:18:32 INFO ParquetOutputFormat: Parquet page size to 1048576
23/09/24 21:18:32 INFO ParquetOutputFormat: Parquet dictionary page size to 1048576
23/09/24 21:18:32 INFO ParquetOutputFormat: Dictionary is on
23/09/24 21:18:32 INFO ParquetOutputFormat: Validation is off
23/09/24 21:18:32 INFO ParquetOutputFormat: Writer version is: PARQUET_1_0
23/09/24 21:18:32 INFO ParquetOutputFormat: Maximum row group padding size is 8388608 bytes
23/09/24 21:18:32 INFO ParquetOutputFormat: Page size checking is: estimated
23/09/24 21:18:32 INFO ParquetOutputFormat: Min row count for page size check is: 100
23/09/24 21:18:32 INFO ParquetOutputFormat: Max row count for page size check is: 10000
23/09/24 21:18:32 INFO ParquetWriteSupport: Initialized Parquet WriteSupport with Catalyst schema:
{
  "type" : "struct",
  "fields" : [ {
    "name" : "z_zid",
    "type" : "long",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "z_cluster",
    "type" : "string",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "z_prediction",
    "type" : "double",
    "nullable" : false,
    "metadata" : {
      "ml_attr" : {
        "type" : "nominal",
        "num_vals" : 2
      }
    }
  }, {
    "name" : "z_score",
    "type" : "double",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "z_isMatch",
    "type" : "integer",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "fname",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "lname",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "stNo",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "add1",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "add2",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "city",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "areacode",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "state",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "dob",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "ssn",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "z_source",
    "type" : "string",
    "nullable" : false,
    "metadata" : { }
  } ]
}
and corresponding Parquet message type:
message spark_schema {
  required int64 z_zid;
  required binary z_cluster (UTF8);
  required double z_prediction;
  optional double z_score;
  required int32 z_isMatch;
  optional binary fname (UTF8);
  optional binary lname (UTF8);
  optional binary stNo (UTF8);
  optional binary add1 (UTF8);
  optional binary add2 (UTF8);
  optional binary city (UTF8);
  optional binary areacode (UTF8);
  optional binary state (UTF8);
  optional binary dob (UTF8);
  optional binary ssn (UTF8);
  required binary z_source (UTF8);
}

       
23/09/24 21:18:32 INFO CodecConfig: Compression: SNAPPY
23/09/24 21:18:32 INFO CodecConfig: Compression: SNAPPY
23/09/24 21:18:32 INFO ParquetOutputFormat: Parquet block size to 134217728
23/09/24 21:18:32 INFO ParquetOutputFormat: Parquet page size to 1048576
23/09/24 21:18:32 INFO ParquetOutputFormat: Parquet dictionary page size to 1048576
23/09/24 21:18:32 INFO ParquetOutputFormat: Dictionary is on
23/09/24 21:18:32 INFO ParquetOutputFormat: Validation is off
23/09/24 21:18:32 INFO ParquetOutputFormat: Writer version is: PARQUET_1_0
23/09/24 21:18:32 INFO ParquetOutputFormat: Maximum row group padding size is 8388608 bytes
23/09/24 21:18:32 INFO ParquetOutputFormat: Page size checking is: estimated
23/09/24 21:18:32 INFO ParquetOutputFormat: Min row count for page size check is: 100
23/09/24 21:18:32 INFO ParquetOutputFormat: Max row count for page size check is: 10000
23/09/24 21:18:32 INFO ParquetWriteSupport: Initialized Parquet WriteSupport with Catalyst schema:
{
  "type" : "struct",
  "fields" : [ {
    "name" : "z_zid",
    "type" : "long",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "z_cluster",
    "type" : "string",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "z_prediction",
    "type" : "double",
    "nullable" : false,
    "metadata" : {
      "ml_attr" : {
        "type" : "nominal",
        "num_vals" : 2
      }
    }
  }, {
    "name" : "z_score",
    "type" : "double",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "z_isMatch",
    "type" : "integer",
    "nullable" : false,
    "metadata" : { }
  }, {
    "name" : "fname",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "lname",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "stNo",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "add1",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "add2",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "city",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "areacode",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "state",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "dob",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "ssn",
    "type" : "string",
    "nullable" : true,
    "metadata" : { }
  }, {
    "name" : "z_source",
    "type" : "string",
    "nullable" : false,
    "metadata" : { }
  } ]
}
and corresponding Parquet message type:
message spark_schema {
  required int64 z_zid;
  required binary z_cluster (UTF8);
  required double z_prediction;
  optional double z_score;
  required int32 z_isMatch;
  optional binary fname (UTF8);
  optional binary lname (UTF8);
  optional binary stNo (UTF8);
  optional binary add1 (UTF8);
  optional binary add2 (UTF8);
  optional binary city (UTF8);
  optional binary areacode (UTF8);
  optional binary state (UTF8);
  optional binary dob (UTF8);
  optional binary ssn (UTF8);
  required binary z_source (UTF8);
}

       
23/09/24 21:18:32 INFO InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 408
23/09/24 21:18:32 INFO InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 435
23/09/24 21:18:32 INFO InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 442
23/09/24 21:18:32 INFO InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 443
23/09/24 21:18:32 INFO InternalParquetRecordWriter: Flushing mem columnStore to file. allocated memory: 434
23/09/24 21:18:32 INFO FileOutputCommitter: Saved output of task 'attempt_202309242118317838551213306550278_0212_m_000019_356' to file:/root/zingg/models/100/trainingData/unmarked/_temporary/0/task_202309242118317838551213306550278_0212_m_000019
23/09/24 21:18:32 INFO SparkHadoopMapRedUtil: attempt_202309242118317838551213306550278_0212_m_000019_356: Committed
23/09/24 21:18:32 INFO Executor: Finished task 19.0 in stage 212.0 (TID 356). 10148 bytes result sent to driver
23/09/24 21:18:32 INFO FileOutputCommitter: Saved output of task 'attempt_202309242118315928267449174482837_0212_m_000016_353' to file:/root/zingg/models/100/trainingData/unmarked/_temporary/0/task_202309242118315928267449174482837_0212_m_000016
23/09/24 21:18:32 INFO SparkHadoopMapRedUtil: attempt_202309242118315928267449174482837_0212_m_000016_353: Committed
23/09/24 21:18:32 INFO Executor: Finished task 16.0 in stage 212.0 (TID 353). 10148 bytes result sent to driver
23/09/24 21:18:32 INFO FileOutputCommitter: Saved output of task 'attempt_202309242118315473955502808280318_0212_m_000015_352' to file:/root/zingg/models/100/trainingData/unmarked/_temporary/0/task_202309242118315473955502808280318_0212_m_000015
23/09/24 21:18:32 INFO SparkHadoopMapRedUtil: attempt_202309242118315473955502808280318_0212_m_000015_352: Committed
23/09/24 21:18:32 INFO Executor: Finished task 15.0 in stage 212.0 (TID 352). 10148 bytes result sent to driver
23/09/24 21:18:32 INFO FileOutputCommitter: Saved output of task 'attempt_202309242118317772041960800284662_0212_m_000018_355' to file:/root/zingg/models/100/trainingData/unmarked/_temporary/0/task_202309242118317772041960800284662_0212_m_000018
23/09/24 21:18:32 INFO SparkHadoopMapRedUtil: attempt_202309242118317772041960800284662_0212_m_000018_355: Committed
23/09/24 21:18:32 INFO Executor: Finished task 18.0 in stage 212.0 (TID 355). 10148 bytes result sent to driver
23/09/24 21:18:32 INFO FileOutputCommitter: Saved output of task 'attempt_202309242118316451165061727897996_0212_m_000017_354' to file:/root/zingg/models/100/trainingData/unmarked/_temporary/0/task_202309242118316451165061727897996_0212_m_000017
23/09/24 21:18:32 INFO SparkHadoopMapRedUtil: attempt_202309242118316451165061727897996_0212_m_000017_354: Committed
23/09/24 21:18:32 INFO Executor: Finished task 17.0 in stage 212.0 (TID 354). 10148 bytes result sent to driver
23/09/24 21:18:32 INFO CoarseGrainedExecutorBackend: Driver commanded a shutdown
23/09/24 21:18:32 ERROR CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM
tdown