librelist archives

« back to archive

gitrecommender

gitrecommender

From:
Ale
Date:
2014-12-02 @ 09:48
Hola,

Ejecute lo siguiente

/opt/hadoop/sbin/start-dfs.sh
/opt/hadoop/sbin/start-yarn.sh

yarn jar ./target/gitrecommender-0.0.1-SNAPSHOT-jar-with-dependencies.jar
../data/linux_git_100k_commits/ input output out

para probar si podia ver el job en yarn

pero tira esto:

14/12/02 06:37:33 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at java.util.HashMap.resize(HashMap.java:580)
        at java.util.HashMap.addEntry(HashMap.java:879)
        at java.util.HashMap.put(HashMap.java:505)
        at net.aprendizajengrande.gitrecommender.db.DB.counts(DB.java:217)
        at
net.aprendizajengrande.gitrecommender.Recommend.main(Recommend.java:69)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:212)

Lo mismo pasa con hadoop jar.

Como puedo aumentar el tamaño del heap?
-- 
Alejandro Peralta
Developer
http://www.machinalis.com/

Re: gitrecommender

From:
Ale
Date:
2014-12-02 @ 15:19
Ahí está, lo hice andar.

El conjunto de herramientas me confunde un poco todavía.

Modifique modifique hadoop-env.sh y agregue:

export HADOOP_CLIENT_OPTS="-Xmx2048m $HADOOP_CLIENT_OPTS"

y lo corri con hadoop jar ...



On Tue, Dec 2, 2014 at 6:48 AM, Ale <aperalta@machinalis.com> wrote:

> Hola,
>
> Ejecute lo siguiente
>
> /opt/hadoop/sbin/start-dfs.sh
> /opt/hadoop/sbin/start-yarn.sh
>
> yarn jar ./target/gitrecommender-0.0.1-SNAPSHOT-jar-with-dependencies.jar
> ../data/linux_git_100k_commits/ input output out
>
> para probar si podia ver el job en yarn
>
> pero tira esto:
>
> 14/12/02 06:37:33 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
>         at java.util.HashMap.resize(HashMap.java:580)
>         at java.util.HashMap.addEntry(HashMap.java:879)
>         at java.util.HashMap.put(HashMap.java:505)
>         at net.aprendizajengrande.gitrecommender.db.DB.counts(DB.java:217)
>         at
> net.aprendizajengrande.gitrecommender.Recommend.main(Recommend.java:69)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at
> 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>
> Lo mismo pasa con hadoop jar.
>
> Como puedo aumentar el tamaño del heap?
> --
> Alejandro Peralta
> Developer
> http://www.machinalis.com/
>



-- 
Alejandro Peralta
Developer
http://www.machinalis.com/

Re: [aprendizajengrande] Re: gitrecommender

From:
Pablo Duboue
Date:
2014-12-02 @ 19:03
Y eso lo podes setear tambien en

conf/mapred-site.xml

  <property>
    <name>mapred.child.java.opts</name>
    <value>-Xmx2048m</value>
  </property>

http://stackoverflow.com/questions/8464048/out-of-memory-error-in-hadoop


On Tue, Dec 2, 2014 at 10:19 AM, Ale <aperalta@machinalis.com> wrote:
> Ahí está, lo hice andar.
>
> El conjunto de herramientas me confunde un poco todavía.
>
> Modifique modifique hadoop-env.sh y agregue:
>
> export HADOOP_CLIENT_OPTS="-Xmx2048m $HADOOP_CLIENT_OPTS"
>
> y lo corri con hadoop jar ...
>
>
>
> On Tue, Dec 2, 2014 at 6:48 AM, Ale <aperalta@machinalis.com> wrote:
>>
>> Hola,
>>
>> Ejecute lo siguiente
>>
>> /opt/hadoop/sbin/start-dfs.sh
>> /opt/hadoop/sbin/star t-yarn.sh
>>
>> yarn jar ./target/gitrecommender-0.0.1-SNAPSHOT-jar-with-dependencies.jar
>> ../data/linux_git_100k_commits/ input output out
>>
>> para probar si podia ver el job en yarn
>>
>> pero tira esto:
>>
>> 14/12/02 06:37:33 WARN util.NativeCodeLoader: Unable to load native-hadoop
>> library for your platform... using builtin-java classes where applicable
>> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
>>         at java.util.HashMap.resize(HashMap.java:580)
>>         at java.util.HashMap.addEntry(HashMap.java:879)
>>         at java.util.HashMap.put(HashMap.java:505)
>>         at net.aprendizajengrande.gitrecommender.db.DB.counts(DB
>> .java:217)
>>         at
>> net.aprendizajengrande.gitrecommender.Recommend.main(Recommend.java:69)
>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>         at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>         at
>> 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>         at java.lang.reflect.Method.invoke(Method.java:606)
>>         at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>>
>> Lo mismo pasa con hadoop jar.
>>
>> Como puedo aumentar el tamaño del heap?
>> --
>> Alejandro Peralta
>> Developer
>> http://www.machinalis.com/
>
>
>
>
> --
> Alejandro Peralta
> Developer
> http://www.machinalis.com/