3.1.3
stderr log page for app-20230915152042-0000/2
Showing 36226 Bytes: 0 - 36226 of 36226
Spark Executor Command: "/usr/java8/jdk1.8.0_202/bin/java" "-cp" "/opt/spark/conf/:/opt/spark/jars/*" "-Xmx8192M" "-Dspark.driver.port=37079" "-Dspark.rpc.message.maxSize=200" "-verbose:gc" "-XX:+PrintGCDetails" "-XX:+PrintGCTimeStamps" "-XX:+HeapDumpOnOutOfMemoryError" "-Xloggc:/tmp/memLog.txt" "-XX:+UseCompressedOops" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://CoarseGrainedScheduler@static.154.188.161.5.clients.your-server.de:37079" "--executor-id" "2" "--hostname" "5.161.188.154" "--cores" "5" "--app-id" "app-20230915152042-0000" "--worker-url" "spark://Worker@5.161.188.154:42679" ======================================== Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 23/09/15 15:20:43 INFO CoarseGrainedExecutorBackend: Started daemon with process name: 5365@entity-resolution 23/09/15 15:20:43 INFO SignalUtils: Registering signal handler for TERM 23/09/15 15:20:43 INFO SignalUtils: Registering signal handler for HUP 23/09/15 15:20:43 INFO SignalUtils: Registering signal handler for INT 23/09/15 15:20:43 WARN Utils: Your hostname, entity-resolution resolves to a loopback address: 127.0.1.1; using 5.161.188.154 instead (on interface eth0) 23/09/15 15:20:43 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address 23/09/15 15:20:43 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 23/09/15 15:20:43 INFO SecurityManager: Changing view acls to: root 23/09/15 15:20:43 INFO SecurityManager: Changing modify acls to: root 23/09/15 15:20:43 INFO SecurityManager: Changing view acls groups to: 23/09/15 15:20:43 INFO SecurityManager: Changing modify acls groups to: 23/09/15 15:20:43 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set() 23/09/15 15:20:44 INFO TransportClientFactory: Successfully created connection to static.154.188.161.5.clients.your-server.de/5.161.188.154:37079 after 58 ms (0 ms spent in bootstraps) 23/09/15 15:20:44 INFO SecurityManager: Changing view acls to: root 23/09/15 15:20:44 INFO SecurityManager: Changing modify acls to: root 23/09/15 15:20:44 INFO SecurityManager: Changing view acls groups to: 23/09/15 15:20:44 INFO SecurityManager: Changing modify acls groups to: 23/09/15 15:20:44 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set() 23/09/15 15:20:44 INFO TransportClientFactory: Successfully created connection to static.154.188.161.5.clients.your-server.de/5.161.188.154:37079 after 2 ms (0 ms spent in bootstraps) 23/09/15 15:20:44 INFO DiskBlockManager: Created local directory at /tmp/spark-ad5f8b12-11db-4a30-8bd0-3fad97d4dcc6/executor-6708c4e2-839d-4182-961f-0ee9940483ee/blockmgr-3eef5a12-c031-48c8-bf67-7512293087cf 23/09/15 15:20:44 INFO MemoryStore: MemoryStore started with capacity 6.1 GiB 23/09/15 15:20:44 INFO CoarseGrainedExecutorBackend: Connecting to driver: spark://CoarseGrainedScheduler@static.154.188.161.5.clients.your-server.de:37079 23/09/15 15:20:44 INFO WorkerWatcher: Connecting to worker spark://Worker@5.161.188.154:42679 23/09/15 15:20:44 INFO TransportClientFactory: Successfully created connection to /5.161.188.154:42679 after 3 ms (0 ms spent in bootstraps) 23/09/15 15:20:44 INFO WorkerWatcher: Successfully connected to spark://Worker@5.161.188.154:42679 23/09/15 15:20:44 INFO ResourceUtils: ============================================================== 23/09/15 15:20:44 INFO ResourceUtils: No custom resources configured for spark.executor. 23/09/15 15:20:44 INFO ResourceUtils: ============================================================== 23/09/15 15:20:44 INFO CoarseGrainedExecutorBackend: Successfully registered with driver 23/09/15 15:20:44 INFO Executor: Starting executor ID 2 on host 5.161.188.154 23/09/15 15:20:44 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 41639. 23/09/15 15:20:44 INFO NettyBlockTransferService: Server created on 5.161.188.154:41639 23/09/15 15:20:44 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy 23/09/15 15:20:44 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(2, 5.161.188.154, 41639, None) 23/09/15 15:20:44 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(2, 5.161.188.154, 41639, None) 23/09/15 15:20:44 INFO BlockManager: Initialized BlockManager: BlockManagerId(2, 5.161.188.154, 41639, None) 23/09/15 15:20:44 INFO Executor: Fetching spark://static.154.188.161.5.clients.your-server.de:37079/jars/postgresql-42.6.0.jar with timestamp 1694791241263 23/09/15 15:20:44 INFO TransportClientFactory: Successfully created connection to static.154.188.161.5.clients.your-server.de/5.161.188.154:37079 after 1 ms (0 ms spent in bootstraps) 23/09/15 15:20:44 INFO Utils: Fetching spark://static.154.188.161.5.clients.your-server.de:37079/jars/postgresql-42.6.0.jar to /tmp/spark-ad5f8b12-11db-4a30-8bd0-3fad97d4dcc6/executor-6708c4e2-839d-4182-961f-0ee9940483ee/spark-eda06d69-865e-4c4e-9588-6c1d27207ac8/fetchFileTemp4603412074999432460.tmp 23/09/15 15:20:44 INFO Utils: Copying /tmp/spark-ad5f8b12-11db-4a30-8bd0-3fad97d4dcc6/executor-6708c4e2-839d-4182-961f-0ee9940483ee/spark-eda06d69-865e-4c4e-9588-6c1d27207ac8/20185424331694791241263_cache to /opt/spark/work/app-20230915152042-0000/2/./postgresql-42.6.0.jar 23/09/15 15:20:44 INFO Executor: Adding file:/opt/spark/work/app-20230915152042-0000/2/./postgresql-42.6.0.jar to class loader 23/09/15 15:20:44 INFO Executor: Fetching spark://static.154.188.161.5.clients.your-server.de:37079/jars/zingg-0.3.4-SNAPSHOT.jar with timestamp 1694791241263 23/09/15 15:20:44 INFO Utils: Fetching spark://static.154.188.161.5.clients.your-server.de:37079/jars/zingg-0.3.4-SNAPSHOT.jar to /tmp/spark-ad5f8b12-11db-4a30-8bd0-3fad97d4dcc6/executor-6708c4e2-839d-4182-961f-0ee9940483ee/spark-eda06d69-865e-4c4e-9588-6c1d27207ac8/fetchFileTemp3307939200892959898.tmp 23/09/15 15:20:44 INFO Utils: Copying /tmp/spark-ad5f8b12-11db-4a30-8bd0-3fad97d4dcc6/executor-6708c4e2-839d-4182-961f-0ee9940483ee/spark-eda06d69-865e-4c4e-9588-6c1d27207ac8/-19624413781694791241263_cache to /opt/spark/work/app-20230915152042-0000/2/./zingg-0.3.4-SNAPSHOT.jar 23/09/15 15:20:44 INFO Executor: Adding file:/opt/spark/work/app-20230915152042-0000/2/./zingg-0.3.4-SNAPSHOT.jar to class loader 23/09/15 15:20:46 INFO CoarseGrainedExecutorBackend: Got assigned task 2 23/09/15 15:20:46 INFO CoarseGrainedExecutorBackend: Got assigned task 5 23/09/15 15:20:46 INFO CoarseGrainedExecutorBackend: Got assigned task 8 23/09/15 15:20:46 INFO CoarseGrainedExecutorBackend: Got assigned task 11 23/09/15 15:20:46 INFO Executor: Running task 5.0 in stage 0.0 (TID 5) 23/09/15 15:20:46 INFO Executor: Running task 2.0 in stage 0.0 (TID 2) 23/09/15 15:20:46 INFO Executor: Running task 11.0 in stage 0.0 (TID 11) 23/09/15 15:20:46 INFO Executor: Running task 8.0 in stage 0.0 (TID 8) 23/09/15 15:20:46 INFO TorrentBroadcast: Started reading broadcast variable 1 with 1 pieces (estimated total size 4.0 MiB) 23/09/15 15:20:46 INFO TransportClientFactory: Successfully created connection to static.154.188.161.5.clients.your-server.de/5.161.188.154:37929 after 1 ms (0 ms spent in bootstraps) 23/09/15 15:20:46 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 7.9 KiB, free 6.1 GiB) 23/09/15 15:20:46 INFO TorrentBroadcast: Reading broadcast variable 1 took 83 ms 23/09/15 15:20:46 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 15.3 KiB, free 6.1 GiB) 23/09/15 15:20:47 INFO CodeGenerator: Code generated in 160.059864 ms 23/09/15 15:20:47 INFO FileScanRDD: Reading File path: file:///root/zing_data_pre/third-ubo-20230604-mapped.csv, range: 46137344-47316767, partition values: [empty row] 23/09/15 15:20:47 INFO FileScanRDD: Reading File path: file:///root/zing_data_pre/third-ubo-20230604-mapped.csv, range: 8388608-12582912, partition values: [empty row] 23/09/15 15:20:47 INFO FileScanRDD: Reading File path: file:///root/zing_data_pre/third-ubo-20230604-mapped.csv, range: 33554432-37748736, partition values: [empty row] 23/09/15 15:20:47 INFO FileScanRDD: Reading File path: file:///root/zing_data_pre/third-ubo-20230604-mapped.csv, range: 20971520-25165824, partition values: [empty row] 23/09/15 15:20:47 INFO CodeGenerator: Code generated in 10.460541 ms 23/09/15 15:20:47 INFO TorrentBroadcast: Started reading broadcast variable 0 with 1 pieces (estimated total size 4.0 MiB) 23/09/15 15:20:47 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 24.9 KiB, free 6.1 GiB) 23/09/15 15:20:47 INFO TorrentBroadcast: Reading broadcast variable 0 took 12 ms 23/09/15 15:20:47 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 394.6 KiB, free 6.1 GiB) 23/09/15 15:20:48 INFO Executor: Finished task 11.0 in stage 0.0 (TID 11). 1886 bytes result sent to driver 23/09/15 15:20:48 INFO Executor: Finished task 2.0 in stage 0.0 (TID 2). 1886 bytes result sent to driver 23/09/15 15:20:48 INFO Executor: Finished task 8.0 in stage 0.0 (TID 8). 1886 bytes result sent to driver 23/09/15 15:20:48 INFO Executor: Finished task 5.0 in stage 0.0 (TID 5). 1886 bytes result sent to driver 23/09/15 15:20:48 INFO CoarseGrainedExecutorBackend: Got assigned task 15 23/09/15 15:20:48 INFO Executor: Running task 2.0 in stage 2.0 (TID 15) 23/09/15 15:20:48 INFO CoarseGrainedExecutorBackend: Got assigned task 18 23/09/15 15:20:48 INFO CoarseGrainedExecutorBackend: Got assigned task 21 23/09/15 15:20:48 INFO Executor: Running task 8.0 in stage 2.0 (TID 21) 23/09/15 15:20:48 INFO CoarseGrainedExecutorBackend: Got assigned task 24 23/09/15 15:20:48 INFO Executor: Running task 11.0 in stage 2.0 (TID 24) 23/09/15 15:20:48 INFO MapOutputTrackerWorker: Updating epoch to 1 and clearing cache 23/09/15 15:20:48 INFO Executor: Running task 5.0 in stage 2.0 (TID 18) 23/09/15 15:20:48 INFO TorrentBroadcast: Started reading broadcast variable 4 with 1 pieces (estimated total size 4.0 MiB) 23/09/15 15:20:48 INFO TransportClientFactory: Successfully created connection to /5.161.188.154:41719 after 1 ms (0 ms spent in bootstraps) 23/09/15 15:20:48 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 16.5 KiB, free 6.1 GiB) 23/09/15 15:20:48 INFO TorrentBroadcast: Reading broadcast variable 4 took 21 ms 23/09/15 15:20:48 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 41.1 KiB, free 6.1 GiB) 23/09/15 15:20:49 INFO CodeGenerator: Code generated in 50.039798 ms 23/09/15 15:20:50 INFO CodeGenerator: Code generated in 48.578979 ms 23/09/15 15:20:50 INFO CodeGenerator: Code generated in 40.326099 ms 23/09/15 15:20:50 INFO FileScanRDD: Reading File path: file:///root/zing_data_pre/third-ubo-20230604-mapped.csv, range: 33554432-37748736, partition values: [empty row] 23/09/15 15:20:50 INFO FileScanRDD: Reading File path: file:///root/zing_data_pre/third-ubo-20230604-mapped.csv, range: 8388608-12582912, partition values: [empty row] 23/09/15 15:20:50 INFO FileScanRDD: Reading File path: file:///root/zing_data_pre/third-ubo-20230604-mapped.csv, range: 20971520-25165824, partition values: [empty row] 23/09/15 15:20:50 INFO FileScanRDD: Reading File path: file:///root/zing_data_pre/third-ubo-20230604-mapped.csv, range: 46137344-47316767, partition values: [empty row] 23/09/15 15:20:50 INFO CodeGenerator: Code generated in 15.785981 ms 23/09/15 15:20:50 INFO TorrentBroadcast: Started reading broadcast variable 3 with 1 pieces (estimated total size 4.0 MiB) 23/09/15 15:20:50 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 24.9 KiB, free 6.1 GiB) 23/09/15 15:20:50 INFO TorrentBroadcast: Reading broadcast variable 3 took 27 ms 23/09/15 15:20:50 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 394.6 KiB, free 6.1 GiB) 23/09/15 15:20:50 INFO CodeGenerator: Code generated in 96.560231 ms 23/09/15 15:20:51 INFO MemoryStore: Block rdd_18_11 stored as values in memory (estimated size 966.3 KiB, free 6.1 GiB) 23/09/15 15:20:51 INFO CodeGenerator: Code generated in 4.508085 ms 23/09/15 15:20:51 INFO CodeGenerator: Code generated in 28.111945 ms 23/09/15 15:20:51 INFO Executor: Finished task 11.0 in stage 2.0 (TID 24). 2451 bytes result sent to driver 23/09/15 15:20:51 INFO MemoryStore: Block rdd_18_2 stored as values in memory (estimated size 3.4 MiB, free 6.1 GiB) 23/09/15 15:20:51 INFO Executor: Finished task 2.0 in stage 2.0 (TID 15). 2451 bytes result sent to driver 23/09/15 15:20:51 INFO MemoryStore: Block rdd_18_8 stored as values in memory (estimated size 3.4 MiB, free 6.1 GiB) 23/09/15 15:20:51 INFO Executor: Finished task 8.0 in stage 2.0 (TID 21). 2451 bytes result sent to driver 23/09/15 15:20:51 INFO MemoryStore: Block rdd_18_5 stored as values in memory (estimated size 3.4 MiB, free 6.1 GiB) 23/09/15 15:20:51 INFO Executor: Finished task 5.0 in stage 2.0 (TID 18). 2451 bytes result sent to driver 23/09/15 15:20:51 INFO CoarseGrainedExecutorBackend: Got assigned task 25 23/09/15 15:20:51 INFO Executor: Running task 0.0 in stage 3.0 (TID 25) 23/09/15 15:20:51 INFO MapOutputTrackerWorker: Updating epoch to 2 and clearing cache 23/09/15 15:20:51 INFO TorrentBroadcast: Started reading broadcast variable 5 with 1 pieces (estimated total size 4.0 MiB) 23/09/15 15:20:51 INFO MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 5.0 KiB, free 6.1 GiB) 23/09/15 15:20:51 INFO TorrentBroadcast: Reading broadcast variable 5 took 8 ms 23/09/15 15:20:51 INFO MemoryStore: Block broadcast_5 stored as values in memory (estimated size 10.1 KiB, free 6.1 GiB) 23/09/15 15:20:51 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 1, fetching them 23/09/15 15:20:51 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:37079) 23/09/15 15:20:51 INFO MapOutputTrackerWorker: Got the output locations 23/09/15 15:20:51 INFO ShuffleBlockFetcherIterator: Getting 12 (720.0 B) non-empty blocks including 4 (240.0 B) local and 8 (480.0 B) host-local and 0 (0.0 B) remote blocks 23/09/15 15:20:51 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 12 ms 23/09/15 15:20:51 INFO TransportClientFactory: Successfully created connection to /5.161.188.154:46665 after 1 ms (0 ms spent in bootstraps) 23/09/15 15:20:51 INFO CodeGenerator: Code generated in 9.435705 ms 23/09/15 15:20:51 INFO Executor: Finished task 0.0 in stage 3.0 (TID 25). 2450 bytes result sent to driver 23/09/15 15:20:52 INFO CoarseGrainedExecutorBackend: Got assigned task 28 23/09/15 15:20:52 INFO Executor: Running task 2.0 in stage 4.0 (TID 28) 23/09/15 15:20:52 INFO CoarseGrainedExecutorBackend: Got assigned task 31 23/09/15 15:20:52 INFO CoarseGrainedExecutorBackend: Got assigned task 34 23/09/15 15:20:52 INFO CoarseGrainedExecutorBackend: Got assigned task 37 23/09/15 15:20:52 INFO Executor: Running task 8.0 in stage 4.0 (TID 34) 23/09/15 15:20:52 INFO Executor: Running task 11.0 in stage 4.0 (TID 37) 23/09/15 15:20:52 INFO TorrentBroadcast: Started reading broadcast variable 6 with 1 pieces (estimated total size 4.0 MiB) 23/09/15 15:20:52 INFO Executor: Running task 5.0 in stage 4.0 (TID 31) 23/09/15 15:20:52 INFO MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 16.8 KiB, free 6.1 GiB) 23/09/15 15:20:52 INFO TorrentBroadcast: Reading broadcast variable 6 took 20 ms 23/09/15 15:20:52 INFO MemoryStore: Block broadcast_6 stored as values in memory (estimated size 43.0 KiB, free 6.1 GiB) 23/09/15 15:20:52 INFO BlockManager: Found block rdd_18_11 locally 23/09/15 15:20:52 INFO BlockManager: Found block rdd_18_5 locally 23/09/15 15:20:52 INFO BlockManager: Found block rdd_18_2 locally 23/09/15 15:20:52 INFO BlockManager: Found block rdd_18_8 locally 23/09/15 15:20:52 INFO CodeGenerator: Code generated in 49.707501 ms 23/09/15 15:20:52 INFO CodeGenerator: Code generated in 22.392471 ms 23/09/15 15:20:52 INFO Executor: Finished task 11.0 in stage 4.0 (TID 37). 2326 bytes result sent to driver 23/09/15 15:20:52 INFO Executor: Finished task 5.0 in stage 4.0 (TID 31). 2326 bytes result sent to driver 23/09/15 15:20:52 INFO Executor: Finished task 2.0 in stage 4.0 (TID 28). 2326 bytes result sent to driver 23/09/15 15:20:52 INFO Executor: Finished task 8.0 in stage 4.0 (TID 34). 2326 bytes result sent to driver 23/09/15 15:20:52 INFO CoarseGrainedExecutorBackend: Got assigned task 39 23/09/15 15:20:52 INFO Executor: Running task 1.0 in stage 5.0 (TID 39) 23/09/15 15:20:52 INFO MapOutputTrackerWorker: Updating epoch to 3 and clearing cache 23/09/15 15:20:52 INFO TorrentBroadcast: Started reading broadcast variable 7 with 1 pieces (estimated total size 4.0 MiB) 23/09/15 15:20:52 INFO MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 9.5 KiB, free 6.1 GiB) 23/09/15 15:20:52 INFO TorrentBroadcast: Reading broadcast variable 7 took 7 ms 23/09/15 15:20:52 INFO MemoryStore: Block broadcast_7 stored as values in memory (estimated size 23.7 KiB, free 6.1 GiB) 23/09/15 15:20:52 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 2, fetching them 23/09/15 15:20:52 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:37079) 23/09/15 15:20:52 INFO MapOutputTrackerWorker: Got the output locations 23/09/15 15:20:52 INFO ShuffleBlockFetcherIterator: Getting 12 (3.1 MiB) non-empty blocks including 4 (934.7 KiB) local and 8 (2.2 MiB) host-local and 0 (0.0 B) remote blocks 23/09/15 15:20:52 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 23/09/15 15:20:52 INFO MemoryStore: Block rdd_35_1 stored as values in memory (estimated size 4.5 MiB, free 6.1 GiB) 23/09/15 15:20:52 INFO CodeGenerator: Code generated in 9.456874 ms 23/09/15 15:20:53 INFO MemoryStore: Block rdd_41_1 stored as values in memory (estimated size 4.5 MiB, free 6.1 GiB) 23/09/15 15:20:53 INFO Executor: Finished task 1.0 in stage 5.0 (TID 39). 3204 bytes result sent to driver 23/09/15 15:20:53 INFO CoarseGrainedExecutorBackend: Got assigned task 45 23/09/15 15:20:53 INFO Executor: Running task 1.0 in stage 8.0 (TID 45) 23/09/15 15:20:53 INFO MapOutputTrackerWorker: Updating epoch to 4 and clearing cache 23/09/15 15:20:53 INFO TorrentBroadcast: Started reading broadcast variable 9 with 1 pieces (estimated total size 4.0 MiB) 23/09/15 15:20:53 INFO MemoryStore: Block broadcast_9_piece0 stored as bytes in memory (estimated size 9.1 KiB, free 6.1 GiB) 23/09/15 15:20:53 INFO TorrentBroadcast: Reading broadcast variable 9 took 8 ms 23/09/15 15:20:53 INFO MemoryStore: Block broadcast_9 stored as values in memory (estimated size 22.0 KiB, free 6.1 GiB) 23/09/15 15:20:53 INFO BlockManager: Found block rdd_41_1 locally 23/09/15 15:20:53 INFO MemoryStore: Block taskresult_45 stored as bytes in memory (estimated size 3.1 MiB, free 6.1 GiB) 23/09/15 15:20:53 INFO Executor: Finished task 1.0 in stage 8.0 (TID 45). 3253512 bytes result sent via BlockManager) 23/09/15 15:20:53 INFO CoarseGrainedExecutorBackend: Got assigned task 48 23/09/15 15:20:53 INFO CoarseGrainedExecutorBackend: Got assigned task 51 23/09/15 15:20:53 INFO Executor: Running task 2.0 in stage 9.0 (TID 48) 23/09/15 15:20:53 INFO Executor: Running task 5.0 in stage 9.0 (TID 51) 23/09/15 15:20:53 INFO CoarseGrainedExecutorBackend: Got assigned task 54 23/09/15 15:20:53 INFO CoarseGrainedExecutorBackend: Got assigned task 57 23/09/15 15:20:53 INFO Executor: Running task 8.0 in stage 9.0 (TID 54) 23/09/15 15:20:53 INFO TorrentBroadcast: Started reading broadcast variable 10 with 1 pieces (estimated total size 4.0 MiB) 23/09/15 15:20:53 INFO Executor: Running task 11.0 in stage 9.0 (TID 57) 23/09/15 15:20:53 INFO CoarseGrainedExecutorBackend: Got assigned task 61 23/09/15 15:20:53 INFO Executor: Running task 2.0 in stage 10.0 (TID 61) 23/09/15 15:20:53 INFO TorrentBroadcast: Started reading broadcast variable 11 with 1 pieces (estimated total size 4.0 MiB) 23/09/15 15:20:53 INFO MemoryStore: Block broadcast_10_piece0 stored as bytes in memory (estimated size 18.7 KiB, free 6.1 GiB) 23/09/15 15:20:53 INFO MemoryStore: Block broadcast_11_piece0 stored as bytes in memory (estimated size 18.1 KiB, free 6.1 GiB) 23/09/15 15:20:53 INFO TorrentBroadcast: Reading broadcast variable 10 took 26 ms 23/09/15 15:20:53 INFO MemoryStore: Block broadcast_10 stored as values in memory (estimated size 50.4 KiB, free 6.1 GiB) 23/09/15 15:20:53 INFO TorrentBroadcast: Reading broadcast variable 11 took 18 ms 23/09/15 15:20:53 INFO MemoryStore: Block broadcast_11 stored as values in memory (estimated size 46.5 KiB, free 6.1 GiB) 23/09/15 15:20:54 INFO BlockManager: Found block rdd_18_2 locally 23/09/15 15:20:54 INFO BlockManager: Found block rdd_18_8 locally 23/09/15 15:20:54 INFO BlockManager: Found block rdd_18_5 locally 23/09/15 15:20:54 INFO CodeGenerator: Code generated in 9.090853 ms 23/09/15 15:20:54 INFO BlockManager: Found block rdd_18_11 locally 23/09/15 15:20:54 INFO MemoryStore: Block rdd_59_11 stored as values in memory (estimated size 417.2 KiB, free 6.1 GiB) 23/09/15 15:20:54 INFO CodeGenerator: Code generated in 14.127397 ms 23/09/15 15:20:54 INFO CodeGenerator: Code generated in 5.849439 ms 23/09/15 15:20:54 INFO MemoryStore: Block rdd_59_2 stored as values in memory (estimated size 1503.2 KiB, free 6.1 GiB) 23/09/15 15:20:54 INFO BlockManager: Found block rdd_59_2 locally 23/09/15 15:20:54 INFO MemoryStore: Block rdd_59_5 stored as values in memory (estimated size 1499.0 KiB, free 6.1 GiB) 23/09/15 15:20:54 INFO MemoryStore: Block rdd_59_8 stored as values in memory (estimated size 1519.7 KiB, free 6.1 GiB) 23/09/15 15:20:54 INFO CodeGenerator: Code generated in 9.640682 ms 23/09/15 15:20:54 INFO Executor: Finished task 11.0 in stage 9.0 (TID 57). 2776 bytes result sent to driver 23/09/15 15:20:54 INFO CoarseGrainedExecutorBackend: Got assigned task 62 23/09/15 15:20:54 INFO Executor: Running task 5.0 in stage 10.0 (TID 62) 23/09/15 15:20:54 INFO BlockManager: Found block rdd_59_5 locally 23/09/15 15:20:54 INFO Executor: Finished task 2.0 in stage 10.0 (TID 61). 2714 bytes result sent to driver 23/09/15 15:20:54 INFO CoarseGrainedExecutorBackend: Got assigned task 64 23/09/15 15:20:54 INFO Executor: Running task 8.0 in stage 10.0 (TID 64) 23/09/15 15:20:54 INFO BlockManager: Found block rdd_59_8 locally 23/09/15 15:20:54 INFO Executor: Finished task 2.0 in stage 9.0 (TID 48). 2776 bytes result sent to driver 23/09/15 15:20:54 INFO CoarseGrainedExecutorBackend: Got assigned task 65 23/09/15 15:20:54 INFO Executor: Running task 11.0 in stage 10.0 (TID 65) 23/09/15 15:20:54 INFO BlockManager: Found block rdd_59_11 locally 23/09/15 15:20:54 INFO Executor: Finished task 5.0 in stage 9.0 (TID 51). 2776 bytes result sent to driver 23/09/15 15:20:54 INFO Executor: Finished task 8.0 in stage 9.0 (TID 54). 2776 bytes result sent to driver 23/09/15 15:20:54 INFO Executor: Finished task 5.0 in stage 10.0 (TID 62). 2671 bytes result sent to driver 23/09/15 15:20:54 INFO Executor: Finished task 11.0 in stage 10.0 (TID 65). 2672 bytes result sent to driver 23/09/15 15:20:54 INFO Executor: Finished task 8.0 in stage 10.0 (TID 64). 2672 bytes result sent to driver 23/09/15 15:28:49 INFO CoarseGrainedExecutorBackend: Got assigned task 72 23/09/15 15:28:49 INFO Executor: Running task 1.0 in stage 13.0 (TID 72) 23/09/15 15:28:49 INFO MapOutputTrackerWorker: Updating epoch to 6 and clearing cache 23/09/15 15:28:49 INFO TorrentBroadcast: Started reading broadcast variable 13 with 1 pieces (estimated total size 4.0 MiB) 23/09/15 15:28:49 INFO MemoryStore: Block broadcast_13_piece0 stored as bytes in memory (estimated size 146.8 KiB, free 6.1 GiB) 23/09/15 15:28:49 INFO TorrentBroadcast: Reading broadcast variable 13 took 5 ms 23/09/15 15:28:49 INFO MemoryStore: Block broadcast_13 stored as values in memory (estimated size 566.1 KiB, free 6.1 GiB) 23/09/15 15:28:49 INFO BlockManager: Found block rdd_35_1 locally 23/09/15 15:28:49 INFO CodeGenerator: Code generated in 44.864867 ms 23/09/15 15:28:49 INFO CodeGenerator: Code generated in 6.472041 ms 23/09/15 15:28:50 INFO Executor: Finished task 1.0 in stage 13.0 (TID 72). 2617 bytes result sent to driver 23/09/15 15:28:50 INFO CoarseGrainedExecutorBackend: Got assigned task 76 23/09/15 15:28:50 INFO CoarseGrainedExecutorBackend: Got assigned task 79 23/09/15 15:28:50 INFO Executor: Running task 0.0 in stage 14.0 (TID 76) 23/09/15 15:28:50 INFO Executor: Running task 3.0 in stage 14.0 (TID 79) 23/09/15 15:28:50 INFO MapOutputTrackerWorker: Updating epoch to 7 and clearing cache 23/09/15 15:28:50 INFO TorrentBroadcast: Started reading broadcast variable 14 with 1 pieces (estimated total size 4.0 MiB) 23/09/15 15:28:50 INFO MemoryStore: Block broadcast_14_piece0 stored as bytes in memory (estimated size 16.7 KiB, free 6.1 GiB) 23/09/15 15:28:50 INFO TorrentBroadcast: Reading broadcast variable 14 took 5 ms 23/09/15 15:28:50 INFO MemoryStore: Block broadcast_14 stored as values in memory (estimated size 42.2 KiB, free 6.1 GiB) 23/09/15 15:28:50 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 6, fetching them 23/09/15 15:28:50 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@static.154.188.161.5.clients.your-server.de:37079) 23/09/15 15:28:50 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 6, fetching them 23/09/15 15:28:50 INFO MapOutputTrackerWorker: Got the output locations 23/09/15 15:28:50 INFO ShuffleBlockFetcherIterator: Getting 4 (2.2 MiB) non-empty blocks including 1 (609.1 KiB) local and 3 (1661.1 KiB) host-local and 0 (0.0 B) remote blocks 23/09/15 15:28:50 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 23/09/15 15:28:50 INFO ShuffleBlockFetcherIterator: Getting 4 (6.2 MiB) non-empty blocks including 1 (1579.8 KiB) local and 3 (4.6 MiB) host-local and 0 (0.0 B) remote blocks 23/09/15 15:28:50 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 23/09/15 15:28:50 INFO MemoryStore: Block rdd_89_0 stored as values in memory (estimated size 2.7 MiB, free 6.1 GiB) 23/09/15 15:28:50 INFO CodeGenerator: Code generated in 11.132137 ms 23/09/15 15:28:50 INFO CodeGenerator: Code generated in 9.43983 ms 23/09/15 15:28:50 INFO CodeGenerator: Code generated in 7.340383 ms 23/09/15 15:28:50 INFO CodeGenerator: Code generated in 4.503011 ms 23/09/15 15:28:50 INFO BlockManager: Found block rdd_89_0 locally 23/09/15 15:28:50 INFO CodeGenerator: Code generated in 20.958528 ms 23/09/15 15:28:50 INFO CodeGenerator: Code generated in 11.554806 ms 23/09/15 15:28:50 INFO CodeGenerator: Code generated in 15.988576 ms 23/09/15 15:28:50 INFO MemoryStore: Block rdd_89_3 stored as values in memory (estimated size 9.9 MiB, free 6.0 GiB) 23/09/15 15:28:50 INFO BlockManager: Found block rdd_89_3 locally 23/09/15 15:28:52 INFO MemoryStore: Block rdd_101_0 stored as values in memory (estimated size 11.9 MiB, free 6.0 GiB) 23/09/15 15:28:52 INFO Executor: Finished task 0.0 in stage 14.0 (TID 76). 3959 bytes result sent to driver 23/09/15 15:30:55 INFO MemoryStore: 18 blocks selected for dropping (38.7 MiB bytes) 23/09/15 15:30:55 INFO BlockManager: Dropping block broadcast_3_piece0 from memory 23/09/15 15:30:55 INFO BlockManager: Writing block broadcast_3_piece0 to disk 23/09/15 15:30:55 INFO BlockManager: Dropping block broadcast_3 from memory 23/09/15 15:30:55 INFO BlockManager: Writing block broadcast_3 to disk 23/09/15 15:30:55 INFO BlockManager: Dropping block rdd_41_1 from memory 23/09/15 15:30:55 INFO BlockManager: Dropping block rdd_18_2 from memory 23/09/15 15:30:55 INFO BlockManager: Dropping block rdd_18_8 from memory 23/09/15 15:30:55 INFO BlockManager: Dropping block rdd_18_5 from memory 23/09/15 15:30:55 INFO BlockManager: Dropping block rdd_18_11 from memory 23/09/15 15:30:55 INFO BlockManager: Dropping block rdd_59_2 from memory 23/09/15 15:30:55 INFO BlockManager: Writing block rdd_59_2 to disk 23/09/15 15:30:55 INFO BlockManager: Dropping block rdd_59_5 from memory 23/09/15 15:30:55 INFO BlockManager: Writing block rdd_59_5 to disk 23/09/15 15:30:55 INFO BlockManager: Dropping block rdd_59_8 from memory 23/09/15 15:30:55 INFO BlockManager: Writing block rdd_59_8 to disk 23/09/15 15:30:55 INFO BlockManager: Dropping block rdd_59_11 from memory 23/09/15 15:30:55 INFO BlockManager: Writing block rdd_59_11 to disk 23/09/15 15:30:55 INFO BlockManager: Dropping block broadcast_13_piece0 from memory 23/09/15 15:30:55 INFO BlockManager: Writing block broadcast_13_piece0 to disk 23/09/15 15:30:55 INFO BlockManager: Dropping block broadcast_13 from memory 23/09/15 15:30:55 INFO BlockManager: Writing block broadcast_13 to disk 23/09/15 15:30:55 INFO BlockManager: Dropping block rdd_35_1 from memory 23/09/15 15:30:55 INFO BlockManager: Dropping block broadcast_14_piece0 from memory 23/09/15 15:30:55 INFO BlockManager: Writing block broadcast_14_piece0 to disk 23/09/15 15:30:55 INFO BlockManager: Dropping block broadcast_14 from memory 23/09/15 15:30:55 INFO BlockManager: Writing block broadcast_14 to disk 23/09/15 15:30:55 INFO BlockManager: Dropping block rdd_89_0 from memory 23/09/15 15:30:55 INFO BlockManager: Writing block rdd_89_0 to disk 23/09/15 15:30:55 INFO BlockManager: Dropping block rdd_89_3 from memory 23/09/15 15:30:55 INFO BlockManager: Writing block rdd_89_3 to disk 23/09/15 15:30:55 INFO MemoryStore: After dropping 18 blocks, free memory is 6.0 GiB 23/09/15 15:31:47 INFO MemoryStore: Will not store rdd_101_3 23/09/15 15:31:47 WARN MemoryStore: Not enough space to cache rdd_101_3 in memory! (computed 4.0 GiB so far) 23/09/15 15:31:47 INFO MemoryStore: Memory use = 11.9 MiB (blocks) + 4.0 GiB (scratch space shared across 1 tasks(s)) = 4.0 GiB. Storage limit = 6.0 GiB. 23/09/15 15:31:47 WARN BlockManager: Persisting block rdd_101_3 to disk instead. 23/09/15 16:20:17 WARN BlockManager: Putting block rdd_101_3 failed due to exception com.esotericsoftware.kryo.KryoException: java.io.IOException: No space left on device Serialization trace: buffers (org.apache.spark.sql.execution.columnar.DefaultCachedBatch). 23/09/15 16:20:17 WARN BlockManager: Block rdd_101_3 could not be removed as it was not found on disk or in memory 23/09/15 16:20:17 ERROR Executor: Exception in task 3.0 in stage 14.0 (TID 79) com.esotericsoftware.kryo.KryoException: java.io.IOException: No space left on device Serialization trace: buffers (org.apache.spark.sql.execution.columnar.DefaultCachedBatch) at com.esotericsoftware.kryo.io.Output.flush(Output.java:188) at com.esotericsoftware.kryo.io.Output.require(Output.java:164) at com.esotericsoftware.kryo.io.Output.writeBytes(Output.java:251) at com.esotericsoftware.kryo.io.Output.writeBytes(Output.java:237) at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ByteArraySerializer.write(DefaultArraySerializers.java:49) at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ByteArraySerializer.write(DefaultArraySerializers.java:38) at com.esotericsoftware.kryo.Kryo.writeObjectOrNull(Kryo.java:629) at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:332) at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.write(DefaultArraySerializers.java:302) at com.esotericsoftware.kryo.Kryo.writeObjectOrNull(Kryo.java:629) at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:86) at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:508) at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:651) at org.apache.spark.serializer.KryoSerializationStream.writeObject(KryoSerializer.scala:259) at org.apache.spark.serializer.SerializationStream.writeAll(Serializer.scala:140) at org.apache.spark.serializer.SerializerManager.dataSerializeStream(SerializerManager.scala:176) at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$3(BlockManager.scala:1432) at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$3$adapted(BlockManager.scala:1430) at org.apache.spark.storage.DiskStore.put(DiskStore.scala:70) at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$1(BlockManager.scala:1430) at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1350) at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1414) at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1237) at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:384) at org.apache.spark.rdd.RDD.iterator(RDD.scala:335) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373) at org.apache.spark.rdd.RDD.iterator(RDD.scala:337) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373) at org.apache.spark.rdd.RDD.iterator(RDD.scala:337) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373) at org.apache.spark.rdd.RDD.iterator(RDD.scala:337) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373) at org.apache.spark.rdd.RDD.iterator(RDD.scala:337) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373) at org.apache.spark.rdd.RDD.iterator(RDD.scala:337) at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52) at org.apache.spark.scheduler.Task.run(Task.scala:131) at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:498) at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:501) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: No space left on device at sun.nio.ch.FileDispatcherImpl.write0(Native Method) at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) at sun.nio.ch.IOUtil.write(IOUtil.java:65) at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:211) at org.apache.spark.storage.CountingWritableChannel.write(DiskStore.scala:339) at java.nio.channels.Channels.writeFullyImpl(Channels.java:78) at java.nio.channels.Channels.writeFully(Channels.java:101) at java.nio.channels.Channels.access$000(Channels.java:61) at java.nio.channels.Channels$1.write(Channels.java:174) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) at com.esotericsoftware.kryo.io.Output.flush(Output.java:186) ... 49 more 23/09/15 16:26:51 INFO CoarseGrainedExecutorBackend: Driver commanded a shutdown 23/09/15 16:26:51 INFO MemoryStore: MemoryStore cleared 23/09/15 16:26:51 INFO BlockManager: BlockManager stopped 23/09/15 16:26:51 INFO ShutdownHookManager: Shutdown hook called 23/09/15 16:26:51 INFO ShutdownHookManager: Deleting directory /t