189 8069 5689
Sqoop执行命令：
成都创新互联专注于神农架林区企业网站建设,响应式网站开发,商城系统网站开发。神农架林区网站建设公司,为神农架林区等地区提供建站服务。全流程定制设计，专业设计，全程项目跟踪，成都创新互联专业和态度为您提供的服务
./sqoop import --connect jdbc:oracle:thin:@10.112.101.251:1621:crmadbmr --username bass_etl --password 75!n!u6J --table DBPMSADM.PD_USERSVC_INFO_00 -m 4 --target-dir /ext/ods/PD_USERSVC_INFO_00/2014071906
执行日志：
crmd3n:/d2_data0/user/ocdc/bin/sqoop-1.4.2-cdh5.2.1/bin> 4 --target-dir /ext/ods/PD_USERSVC_INFO_00/2014071906 <
14/07/19 15:31:30 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
14/07/19 15:31:30 INFO manager.SqlManager: Using default fetchSize of 1000
14/07/19 15:31:30 INFO tool.CodeGenTool: Beginning code generation
14/07/19 15:31:31 INFO manager.OracleManager: Time zone has been set to GMT
14/07/19 15:31:31 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM DBPMSADM.PD_USERSVC_INFO_00 t WHERE 1=0
14/07/19 15:31:31 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /d2_data0/user/ocdc/bin/hadoop-2.0.0-mr1-cdh5.2.1
注意：/tmp/sqoop-ocdc/compile/f2f2da5cca8a7ec0811e1390418222a8/DBPMSADM_PD_USERSVC_INFO_00.java 使用或覆盖了已过时的 API。
注意：要了解详细信息，请使用 -Xlint:deprecation 重新编译。
14/07/19 15:31:33 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-ocdc/compile/f2f2da5cca8a7ec0811e1390418222a8/DBPMSADM.PD_USERSVC_INFO_00.jar
14/07/19 15:31:33 INFO manager.OracleManager: Time zone has been set to GMT
14/07/19 15:31:33 WARN manager.OracleManager: The table DBPMSADM.PD_USERSVC_INFO_00 contains a multi-column primary key. Sqoop will default to the column SVC_ID only for this job.
14/07/19 15:31:33 INFO manager.OracleManager: Time zone has been set to GMT
14/07/19 15:31:33 WARN manager.OracleManager: The table DBPMSADM.PD_USERSVC_INFO_00 contains a multi-column primary key. Sqoop will default to the column SVC_ID only for this job.
14/07/19 15:31:33 INFO mapreduce.ImportJobBase: Beginning import of DBPMSADM.PD_USERSVC_INFO_00
14/07/19 15:31:33 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/07/19 15:31:33 INFO manager.OracleManager: Time zone has been set to GMT
14/07/19 15:31:34 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
14/07/19 15:31:36 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(SVC_ID), MAX(SVC_ID) FROM DBPMSADM.PD_USERSVC_INFO_00
14/07/19 15:31:37 WARN db.TextSplitter: Generating splits for a textual index column.
14/07/19 15:31:37 WARN db.TextSplitter: If your database sorts in a case-insensitive order, this may result in a partial import or duplicate records.
14/07/19 15:31:37 WARN db.TextSplitter: You are strongly encouraged to choose an integral split column.
14/07/19 15:31:38 INFO mapred.JobClient: Running job: job_201407151514_3489
14/07/19 15:31:39 INFO mapred.JobClient: map 0% reduce 0%
14/07/19 15:31:50 INFO mapred.JobClient: Task Id : attempt_201407151514_3489_m_000000_0, Status : FAILED
java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:70)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:636)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection
14/07/19 15:31:51 INFO mapred.JobClient: Task Id : attempt_201407151514_3489_m_000001_0, Status : FAILED
java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:70)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:636)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection
14/07/19 15:31:51 INFO mapred.JobClient: Task Id : attempt_201407151514_3489_m_000003_0, Status : FAILED
java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:70)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:636)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection
14/07/19 15:31:53 INFO mapred.JobClient: map 25% reduce 0%
14/07/19 15:31:58 INFO mapred.JobClient: Task Id : attempt_201407151514_3489_m_000001_1, Status : FAILED
java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:70)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:636)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection
14/07/19 15:31:58 INFO mapred.JobClient: Task Id : attempt_201407151514_3489_m_000000_1, Status : FAILED
java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:70)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:636)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection
14/07/19 15:31:59 INFO mapred.JobClient: Task Id : attempt_201407151514_3489_m_000003_1, Status : FAILED
java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:70)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:636)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection
14/07/19 15:32:05 INFO mapred.JobClient: Task Id : attempt_201407151514_3489_m_000000_2, Status : FAILED
java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:70)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:636)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection
14/07/19 15:32:06 INFO mapred.JobClient: Task Id : attempt_201407151514_3489_m_000003_2, Status : FAILED
java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection
at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:70)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:636)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.RuntimeException: java.sql.SQLException: Io exception: The Network Adapter could not establish the connection
at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection
14/07/19 15:32:08 INFO mapred.JobClient: map 50% reduce 0%
14/07/19 15:32:17 INFO mapred.JobClient: Job complete: job_201407151514_3489
14/07/19 15:32:17 INFO mapred.JobClient: Counters: 24
14/07/19 15:32:17 INFO mapred.JobClient: File System Counters
14/07/19 15:32:17 INFO mapred.JobClient: FILE: Number of bytes read=0
14/07/19 15:32:17 INFO mapred.JobClient: FILE: Number of bytes written=369072
14/07/19 15:32:17 INFO mapred.JobClient: FILE: Number of read operations=0
14/07/19 15:32:17 INFO mapred.JobClient: FILE: Number of large read operations=0
14/07/19 15:32:17 INFO mapred.JobClient: FILE: Number of write operations=0
14/07/19 15:32:17 INFO mapred.JobClient: HDFS: Number of bytes read=256
14/07/19 15:32:17 INFO mapred.JobClient: HDFS: Number of bytes written=8
14/07/19 15:32:17 INFO mapred.JobClient: HDFS: Number of read operations=2
14/07/19 15:32:17 INFO mapred.JobClient: HDFS: Number of large read operations=0
14/07/19 15:32:17 INFO mapred.JobClient: HDFS: Number of write operations=2
14/07/19 15:32:17 INFO mapred.JobClient: Job Counters
14/07/19 15:32:17 INFO mapred.JobClient: Failed map tasks=1
14/07/19 15:32:17 INFO mapred.JobClient: Launched map tasks=12
14/07/19 15:32:17 INFO mapred.JobClient: Total time spent by all maps in occupied slots (ms)=91182
14/07/19 15:32:17 INFO mapred.JobClient: Total time spent by all reduces in occupied slots (ms)=0
14/07/19 15:32:17 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
14/07/19 15:32:17 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
14/07/19 15:32:17 INFO mapred.JobClient: Map-Reduce Framework
14/07/19 15:32:17 INFO mapred.JobClient: Map input records=0
14/07/19 15:32:17 INFO mapred.JobClient: Map output records=0
14/07/19 15:32:17 INFO mapred.JobClient: Input split bytes=256
14/07/19 15:32:17 INFO mapred.JobClient: Spilled Records=0
14/07/19 15:32:17 INFO mapred.JobClient: CPU time spent (ms)=8910
14/07/19 15:32:17 INFO mapred.JobClient: Physical memory (bytes) snapshot=698327040
14/07/19 15:32:17 INFO mapred.JobClient: Virtual memory (bytes) snapshot=7732314112
14/07/19 15:32:17 INFO mapred.JobClient: Total committed heap usage (bytes)=2022703104
14/07/19 15:32:17 INFO mapreduce.ImportJobBase: Transferred 8 bytes in 43.3403 seconds (0.1846 bytes/sec)
14/07/19 15:32:17 INFO mapreduce.ImportJobBase: Retrieved 0 records.
14/07/19 15:32:17 ERROR tool.ImportTool: Error during import: Import job failed!
昨天发了从DB2中导入数据的错误日志，后来定位为JDBC驱动包的问题，但是替换之后还是没用，然后又定位为是由于JDBC线程未关闭，然后换成了从Oracle中导入数据，又来了另外一个问题：连接有时能通，有时不能通，对于-m 4是基本通不了，原因在于Mapreduce是自动分发匹配的，所以有些结点可能是不可见的，这样对我们连接又会出现网络不通的问题，初步解决方案：自行修改源代码，找到任务分发代码，直接指定一个专门用来做数据导入的结点。正在实践ing。。。。