Lesson for tonight: make -j is a bad idea

Discussion in 'OT Technology' started by Penguin Man, Jun 12, 2006.

  1. Penguin Man

    Penguin Man Protect Your Digital Liberties

    Joined:
    Apr 27, 2002
    Messages:
    21,696
    Likes Received:
    0
    Location:
    Edmonton, AB
    Several weeks ago, I was compiling Firefox and stumbled accross the magic that is make -j. The -j n flag tells make to run n processes at once, and if you leave n out, it runs as many as it can. Now, I compiled Firefox with make -j and it went along just fine - super ultra fast on my new Athlon64 X2 system. In my moment of happiness, I aliased make to make -j, so I wouldn't need to remember about this great trick in the future.

    So tonight I go to compile Gnash, the GNU Flash replacement (which sucks, BTW) and I configure it and type make, expecting it to go along its merry way. Which it does, for a while. Then my computer completely hangs up, the console tells me it's killing processes because I have no memory left and I'm going :wtf:

    After rebooting, I go to compile again, and sure enough, it slows right down again. So I kill it and try again while running top in another window. 320 processes :eek3: Kill the compile and it goes down to 140.

    Turned it down to make -j 8 and all went well.

    So, lesson: make -j really will run as many processes as it can. And my computer doesn't like 150 instances of g++ running at once.
     
  2. Bruticus

    Bruticus half dead OT Supporter

    Joined:
    Apr 10, 2004
    Messages:
    4,608
    Likes Received:
    0
    Location:
    Melbourne
    :rofl: Not surprising :) It really shouldn't work without you specifying a limit, or it having an inbuilt one.
     
  3. deusexaethera

    deusexaethera OT Supporter

    Joined:
    Jan 27, 2005
    Messages:
    19,712
    Likes Received:
    0
    That's nuts. Y'know, I almost like the idea of compiling all my programs on my own computer, just because I could be sure it would run correctly, but for now the convenience of Windows' bloated compatibility interface is just more convenient. Linux is about two years away from being user-friendly enough for me to dive into, and it's been there since about 1998. (shrug)
     
  4. P07r0457

    P07r0457 New Member

    Joined:
    Sep 20, 2004
    Messages:
    28,491
    Likes Received:
    0
    Location:
    Southern Oregon
    it won't change much... if it got to where you need it to be, most linux advocates would hate it. It's benefits are level of customization it offers -- and customization on this scale will always take away from convenience.
     
  5. SLED

    SLED build an idiot proof device and someone else will

    Joined:
    Sep 20, 2001
    Messages:
    28,118
    Likes Received:
    0
    Location:
    AZ, like a bauce!
    distcc FTW :o
     
  6. deusexaethera

    deusexaethera OT Supporter

    Joined:
    Jan 27, 2005
    Messages:
    19,712
    Likes Received:
    0
    That is quite true. If only it weren't so convenient to maintain the status quo, I would be more adventurous. It would be nice, at least, if a .NET framework existed for Linux; it would give me a reason to migrate all my old code to VS.NET, so I could run it on Linux without having to transcode it all to STL C++.
     
  7. SLED

    SLED build an idiot proof device and someone else will

    Joined:
    Sep 20, 2001
    Messages:
    28,118
    Likes Received:
    0
    Location:
    AZ, like a bauce!
    There is Mono if you are looking for a .NET like framework on Linux. Problem is, it is usually significantly behind MS.NET in functionality. I have built a few apps, just for myself, but nothing production yet.
     
  8. P07r0457

    P07r0457 New Member

    Joined:
    Sep 20, 2004
    Messages:
    28,491
    Likes Received:
    0
    Location:
    Southern Oregon
    I equate Mono to the old "chilisoft". Cool in principle, but horrible in execution.

    nothing more than a novel toy.
     
  9. deusexaethera

    deusexaethera OT Supporter

    Joined:
    Jan 27, 2005
    Messages:
    19,712
    Likes Received:
    0
    Hence the problem. If only someone could create a VB -> Java transcoder that actually works; I tried one a few years back, but it too failed miserably.
     
  10. P07r0457

    P07r0457 New Member

    Joined:
    Sep 20, 2004
    Messages:
    28,491
    Likes Received:
    0
    Location:
    Southern Oregon
    transcoders will probably never work well.

    Programming languages are quite different. Much how verbal languages are different. Word-for-word translations can get the basic point across, but each has nuances. The only proper way to switch languages would be to re-write it natively.
     
  11. Penguin Man

    Penguin Man Protect Your Digital Liberties

    Joined:
    Apr 27, 2002
    Messages:
    21,696
    Likes Received:
    0
    Location:
    Edmonton, AB
    Yeah, we were playing with it at school this past year. It's very cool.

    On the Linux topic, I had a chance to attend a Q&A session with Ken Thompson this morning. He feels that Linux has moved too far towards being a Windows replacement, at the cost of elegance in the kernel. In his experience, the Linux kernel has become such a kludge that it is now impossible to remove a feature you don't want without breaking everything else. Very interesting perspective.
     
  12. deusexaethera

    deusexaethera OT Supporter

    Joined:
    Jan 27, 2005
    Messages:
    19,712
    Likes Received:
    0
    Indeed. That's why things like .NET exist in the first place; to serve as a common basis for multiple high-level languages. I just really don't want to bother with adapting my VB6 code to .NET just so I could try to run it on a linux OS with a .Nockoff.
     
  13. Peyomp

    Peyomp New Member

    Joined:
    Jan 11, 2002
    Messages:
    14,017
    Likes Received:
    0
    They need to institute a feature freeze, and just fix bugs/refactor for 6 months or something. They should do this right after they bring Reiser4 into the main line.
     
  14. Peyomp

    Peyomp New Member

    Joined:
    Jan 11, 2002
    Messages:
    14,017
    Likes Received:
    0
    So Gentoo docs indicate that make -jX is the most efficient, where X is the number of processors on your system... so, I've got a Core Duo so I'm thinking two is the right number but... I'm running this through Parallels VM stuff. So the question is: what make -jX is most efficient if you're using Core Duo's virtualization optimizations? 1 or 2?
     
  15. P07r0457

    P07r0457 New Member

    Joined:
    Sep 20, 2004
    Messages:
    28,491
    Likes Received:
    0
    Location:
    Southern Oregon
    at least 2. I know with a HT CPU you go with 2.
     
  16. Peyomp

    Peyomp New Member

    Joined:
    Jan 11, 2002
    Messages:
    14,017
    Likes Received:
    0
    I went with 2.
     
  17. SLED

    SLED build an idiot proof device and someone else will

    Joined:
    Sep 20, 2001
    Messages:
    28,118
    Likes Received:
    0
    Location:
    AZ, like a bauce!
    2 fo' shizzle on a Core Duo... assuming parallels exposes both procs. what does 'cat /proc/cpuinfo' show you?
     
  18. Penguin Man

    Penguin Man Protect Your Digital Liberties

    Joined:
    Apr 27, 2002
    Messages:
    21,696
    Likes Received:
    0
    Location:
    Edmonton, AB
    I find 8 works just fine on my X2 3800+.
     
  19. Peyomp

    Peyomp New Member

    Joined:
    Jan 11, 2002
    Messages:
    14,017
    Likes Received:
    0
    processor 0
    vendor_is GenuineIntel
    cpu_family 6
    model 6
    model name Celeron-A
    stepping 5
    cpu MHz 45667.336
    cache size 2048 MB
    fpu yes
    fpu_exception yes
    cpuid level 2
    wp yes
    bogomips 2387.86

    Some other info, but that is most of it.
     
  20. SLED

    SLED build an idiot proof device and someone else will

    Joined:
    Sep 20, 2001
    Messages:
    28,118
    Likes Received:
    0
    Location:
    AZ, like a bauce!
    yea, so after installing Parallels last night, I found the same thing. Looks like they only expose 1 virtual processor instead of two.

    EDIT: Wait, celeron????
     
  21. Peyomp

    Peyomp New Member

    Joined:
    Jan 11, 2002
    Messages:
    14,017
    Likes Received:
    0
    Is a Core Duo a celeron type?
     
  22. SLED

    SLED build an idiot proof device and someone else will

    Joined:
    Sep 20, 2001
    Messages:
    28,118
    Likes Received:
    0
    Location:
    AZ, like a bauce!
    fuck no
     
  23. deusexaethera

    deusexaethera OT Supporter

    Joined:
    Jan 27, 2005
    Messages:
    19,712
    Likes Received:
    0
    uh...Core Duos don't have two virtual cores, they have two actual cores. The cores don't pretend to be separate, as is the case with a P4 w/HyperThreading. They actually are separate, they cannot be recombined like HT's virtual cores, they just share the same L1 cache. So the answer is, run "make -J2".
     
  24. SLED

    SLED build an idiot proof device and someone else will

    Joined:
    Sep 20, 2001
    Messages:
    28,118
    Likes Received:
    0
    Location:
    AZ, like a bauce!
    yes, thanks for the breaking news. If you actually READ the thread then you would know that he is running a virtual pc emulator called Parallels on his MBP and he has installed gentoo in an instance of this software. Gentoo is reporting 1 virtual processor.
     
  25. deusexaethera

    deusexaethera OT Supporter

    Joined:
    Jan 27, 2005
    Messages:
    19,712
    Likes Received:
    0
    I assume you mean this:

    Obviously I misunderstood what Peyomp said. I didn't know that Parallels was a virtual machine framework for *nix; knowing that would have helped my comprehension, but what I got wrapped around the axle on was the part about "Core Duo's virtualization options" -- note that it didn't say "Parallels' virtualization options" -- hence my reply that Core Duo has multiple real cores instead of virtual ones like the HT Pentiums that Jolly referred to at one point. My bad.
     

Share This Page