Coding: Fleeting Thoughts

A place to discuss the implementation and style of computer programs.

Moderators: phlip, Moderators General, Prelates

EvanED
Posts: 4324
Joined: Mon Aug 07, 2006 6:28 am UTC
Location: Madison, WI
Contact:

Re: Coding: Fleeting Thoughts

Postby EvanED » Wed Jul 20, 2016 1:28 am UTC

korona wrote:Graph data structures need to be tightly coupled to the problem they want to solve. Even something simple like DFS requires specialized data structures (i.e. a per-node flag that remembers if the node has already been visited yet).
You wouldn't need to store that flag intrusively though. It might be necessary for a really-high-performance library, but lots of problems are small.

korona
Posts: 495
Joined: Sun Jul 04, 2010 8:40 pm UTC

Re: Coding: Fleeting Thoughts

Postby korona » Wed Jul 20, 2016 7:44 am UTC

Are there that many non-toy applications that use graphs but do not need high performance OR specialized structures? And for those applications that use small graphs, is writing two small classes that much overhead?

By the way: Constructing high performance graph data structures is hard. I've worked on a project that needed to represent large non-immutable hypergraphs before. Here are some optimizations we had to use:

  • Edges were stored as a head (containing number of nodes + flags) + an array of their incident nodes stored continuously in a large buffer. Deleted edges where marked with a flag. As soon as the deleted edges used up more than 50% of the buffer we went trough the buffer and garbage collected the edges. Pointers to edges in other parts of the program had to be adjusted afterwards.
  • Instead of using native pointers for nodes / edges we used 31-bit integers. This limited us to 2 billion nodes / edges but also halved the cache footprint on 64-bit machines.
  • The remaining bit was used in adjacency lists to distinguish 2-edges from k-edges. Adjacency lists consisted of two 32-bit integers: The first one identified the edge (plus the 2-edge/k-edge bit) and the second one identified the first node of the edge. This had two advantages: For 2-edges the edge buffer did not need to be inspected while traversing the graph and for k-edges we could prefetch the relevant cache line from the edge buffer while traversing the first node of the edge.

EvanED
Posts: 4324
Joined: Mon Aug 07, 2006 6:28 am UTC
Location: Madison, WI
Contact:

Re: Coding: Fleeting Thoughts

Postby EvanED » Wed Jul 20, 2016 8:44 pm UTC

korona wrote:Are there that many non-toy applications that use graphs but do not need high performance OR specialized structures?
First, I'm not sure there are "that many", but I personally fairly frequently do one thing or another on a reasonably small graph (hundreds to hundreds of thousands of nodes) in Python. I'm not sure how many times I've written a depth-first search because there isn't one; if I had a dollar for each time, I'd have.. well, more dollars than I want for that reason. :-)

And that I think feeds into your second point. I'm not sure how much special data structures are required. I can imagine that a useful Graph library could be created that doesn't provide any data structures, just algorithms. You could represent your graph in whatever way you want.

Which is also why I don't really like this argument:

And for those applications that use small graphs, is writing two small classes that much overhead?


It's not two small classes, in fact, like I said I'm not sure you need any classes. (And in fact, in my idealized "little graph library", I wouldn't even need to convert my representation to whatever the library wants.) It's the various algorithms that you can apply to graphs.

User avatar
Xanthir
My HERO!!!
Posts: 5214
Joined: Tue Feb 20, 2007 12:49 am UTC
Location: The Googleplex
Contact:

Re: Coding: Fleeting Thoughts

Postby Xanthir » Sat Jul 23, 2016 3:35 am UTC

Yeah, I've got some relatively common use-cases throughout my codebase where I have graphs of things, and I could really use the ability to tell if there's a path between two points, and sometimes what that path is. While perf is always important, these aren't hot mathematical loops or anything.
(defun fibs (n &optional (a 1) (b 1)) (take n (unfold '+ a b)))

korona
Posts: 495
Joined: Sun Jul 04, 2010 8:40 pm UTC

Re: Coding: Fleeting Thoughts

Postby korona » Sat Jul 23, 2016 9:57 am UTC

In C++ does

Code: Select all

struct B {
    virtual void foo() = 0;
};
struct A : public B {
    void foo() override;
} // no data members

imply
sizeof(A) == sizeof(B)? I guess it does in all sensible ABIs but I suspect it is not enforced by the standard.

User avatar
Yakk
Poster with most posts but no title.
Posts: 11045
Joined: Sat Jan 27, 2007 7:27 pm UTC
Location: E pur si muove

Re: Coding: Fleeting Thoughts

Postby Yakk » Sat Jul 23, 2016 3:13 pm UTC

Once you add virtual or a few other things, C++ does *not* provide layout/size guarantees.

Its layout/size guarantees exist basically to be compatible with C's layout/size guarantees. There has been merely modest expansion on top of that, allowing things that should have zero impact on the C-style layout to have a guarantee.

A virtual function table is not a C-style layout, so nope, no guarantees. They could encode the objects vtable inline if the compiler wanted to.
One of the painful things about our time is that those who feel certainty are stupid, and those with any imagination and understanding are filled with doubt and indecision - BR

Last edited by JHVH on Fri Oct 23, 4004 BCE 6:17 pm, edited 6 times in total.

korona
Posts: 495
Joined: Sun Jul 04, 2010 8:40 pm UTC

Re: Coding: Fleeting Thoughts

Postby korona » Sat Jul 23, 2016 9:58 pm UTC

Well I'm not interested in the size of the structs compared to C types but only to each other (and their vtables would be of the same size). I guess Itanium ABI guarantees that they are the same size so I'll just write my code under this assumption and add a static_assert so that other people can fix it if it does not work for their ABI.

User avatar
Yakk
Poster with most posts but no title.
Posts: 11045
Joined: Sat Jan 27, 2007 7:27 pm UTC
Location: E pur si muove

Re: Coding: Fleeting Thoughts

Postby Yakk » Mon Jul 25, 2016 3:15 pm UTC

I am not certain what good comes of your plan. Do you plan on using the storage declared for an `A` and destroying it, and storing a `B` there? That will result in undefined behavior 3 ways from sunday.

If you are, say, trying to create polymorphic stack (automatic storage) objects, store an A* and a buffer with some alignment and size. Static assert that the type T you are constructing fits in the alignment and size ((alignof(T)/alignof(buffer)-1*alignof(buffer)-1 + sizeof(T) <= sizeof buffer, say, or maybe that the T has bounded alignment and size by the buffer size which is less complex and 99.9% of the way there). Use the aligned storage APIs in std to get a pointer suitablly aligned for a type T, where T is a descendent class of A, in the buffer, placement new it, and set the A* pointer to point to the return value of placement new.

And none of this assumes that your B is the exact same size as your A.

If you make your poly_auto<A, size, align> type be poly_auto<A, sizeof(A), alignof(A)>, then only types with the same size/alignment or less can fit in it, but the code doesn't fundamentally assume it. This also means that the poly_auto object can be made bigger to permit bigger descendents, which is quite useful!

But maybe you have a different plan. (What you are saying looks a bit like what I thought about when I ended up with a poly_auto type)
One of the painful things about our time is that those who feel certainty are stupid, and those with any imagination and understanding are filled with doubt and indecision - BR

Last edited by JHVH on Fri Oct 23, 4004 BCE 6:17 pm, edited 6 times in total.

User avatar
Link
Posts: 1327
Joined: Sat Mar 07, 2009 11:33 am UTC
Location: ᘝᓄᘈᖉᐣ
Contact:

Re: Coding: Fleeting Thoughts

Postby Link » Mon Jul 25, 2016 4:03 pm UTC

korona wrote:
Well I'm not interested in the size of the structs compared to C types but only to each other (and their vtables would be of the same size). I guess Itanium ABI guarantees that they are the same size so I'll just write my code under this assumption and add a static_assert so that other people can fix it if it does not work for their ABI.

So what you're basically saying is "I'm going to rely on undefined behaviour being implemented in one particular way, and if it breaks for someone else, that's their problem"? Because that's not a very friendly thing to do, TBH. Plus, you're leaving the window open for all manner of nasty bugs to come in and bite you later. If you're coding for an embedded device where everything is specific to the machine you're using, I suppose it's not too bad, but if not, you're setting yourself up to (1) be shot in the foot at some point in the future and (2) get loads of hate-mail from people who unexpectedly trip the static_assert.

korona
Posts: 495
Joined: Sun Jul 04, 2010 8:40 pm UTC

Re: Coding: Fleeting Thoughts

Postby korona » Mon Jul 25, 2016 5:32 pm UTC

Yakk wrote:If you make your poly_auto<A, size, align> type be poly_auto<A, sizeof(A), alignof(A)>, then only types with the same size/alignment or less can fit in it, but the code doesn't fundamentally assume it. This also means that the poly_auto object can be made bigger to permit bigger descendents, which is quite useful!

But maybe you have a different plan. (What you are saying looks a bit like what I thought about when I ended up with a poly_auto type)

Yeah that is basically what I am doing. I need to implement a callback by overriding a virtual method of some base class and passing it to a library (that I don't control). I already have an object X that has a lifetime that encloses that callbacks lifetime, so I don't want to allocate the callback on the heap. Unfortunately the derived class must be a template so I cannot just stuff an object of the derived class into X (which is not a template).

Link wrote:
korona wrote:
Well I'm not interested in the size of the structs compared to C types but only to each other (and their vtables would be of the same size). I guess Itanium ABI guarantees that they are the same size so I'll just write my code under this assumption and add a static_assert so that other people can fix it if it does not work for their ABI.

So what you're basically saying is "I'm going to rely on undefined behaviour being implemented in one particular way, and if it breaks for someone else, that's their problem"?

I'm not relying on undefined behavior. I'm relying on behavior that is defined in the ABI and not in the C++ standard. There is a huge difference between undefined and implementation defined behavior. If you ever called a POSIX or WinAPI function you're doing exactly the same.

I'm also not exposing myself to unexpected bugs. Either the static_assert triggers and prevents the code from compiling or the code runs fine (as the correctness at runtime depends on the C++ standard only).

User avatar
Yakk
Poster with most posts but no title.
Posts: 11045
Joined: Sat Jan 27, 2007 7:27 pm UTC
Location: E pur si muove

Re: Coding: Fleeting Thoughts

Postby Yakk » Mon Jul 25, 2016 7:46 pm UTC

I am uncertain, but the issue is you are taking an object of type A and replacing it with an object of type B.

This may violate aliasing requirements unless you are very careful.

If you are violating aliasing, this is UB. One way the compiler could "exploit" that UB in a way that leads to pain for you is it may prove that all well-formed programs have the pointer they have pointing at a value of actual type A, not a value of type B.

Once that proof has been done, the compiler can devirtualize the method calls. Which bypasses your ABI-based virtual function pointer replacement.

The devirtualization could be as simple as "well, I just noticed this was type A, and I cached the vtable lookup. Now your code runs and it replaces that A with a B. I don't bother to look at the vtable, because *no well formed program could have replaced the A with a B* during that interval.", or somesuch.

Now, this may not be a concern. I'm not an expert on aliasing requirements, or what happens when you take a non-standard-layout object and destroy it and construct a new one in its place with regards to the standard. Mostly because it isn't hard to do it strictly correctly, and I like having code that 10 years later won't explode/corrupt memory/misbehave in an otherwise innocuous compiler update.

But here is a non-expert guess:

I mean, there are subtle rules about how long an object lives. And if you have a pointer to an object that is no longer alive, dereferencing it is UB. Creating a new object in the same location does not, I believe, rescue the pointer. So code with an A* pointing to an object is freely allowed to assume that the type of the object it is pointing to *does not change* so long as the pointer does not change, permitting caching of vtable lookups and the like. Thus the strategy of destroying and object and creating a new one invalidates all pointers to the object, and it working is just a slightly more reliable version of

Code: Select all

int * x = new int(3);
delete x;
new int(4);
std::cout << *x;

printing 4.

However, the best part?

Code: Select all

    int * x = new int(3);
    delete x;
    int * y = new int(4);
    std::cout << x << ":" << *x << "," << y << ":" << *y;

prints "0xf46c20:0,0xf46c20:4". The gcc compiler *did not dereferene x* in the above code.

Clang, meanwhile, does dereference x, but not if I just print "std::cout << *x <<"\n";", then it skips reading x.
One of the painful things about our time is that those who feel certainty are stupid, and those with any imagination and understanding are filled with doubt and indecision - BR

Last edited by JHVH on Fri Oct 23, 4004 BCE 6:17 pm, edited 6 times in total.

Tub
Posts: 310
Joined: Wed Jul 27, 2011 3:13 pm UTC

Re: Coding: Fleeting Thoughts

Postby Tub » Mon Jul 25, 2016 8:44 pm UTC

korona wrote:I need to implement a callback by overriding a virtual method of some base class and passing it to a library (that I don't control).

Have you considered using something as simple as a function pointer? Guaranteed layout and ABI stability; little overhead; replacable anytime you like. Much safer than trying to change vtables.

Code: Select all

struct A : public B {
    A() : callback(nullptr) {}

    void foo() override {
        if (callback != nullptr)
            (*callback)();
    }
    void (*callback)(void);
}

A myA;
myA.callback = bar;

korona
Posts: 495
Joined: Sun Jul 04, 2010 8:40 pm UTC

Re: Coding: Fleeting Thoughts

Postby korona » Mon Jul 25, 2016 11:03 pm UTC

Yakk wrote:I am uncertain, but the issue is you are taking an object of type A and replacing it with an object of type B.

This may violate aliasing requirements unless you are very careful.

Ah, I think you misunderstood me. I don't want to do that. I'm just assuming that B will fit into an std::aligned_storage<sizeof(A)> and I static_assert that it actually does.

Tub wrote:
korona wrote:I need to implement a callback by overriding a virtual method of some base class and passing it to a library (that I don't control).

Have you considered using something as simple as a function pointer? Guaranteed layout and ABI stability; little overhead; replacable anytime you like. Much safer than trying to change vtables.

I actually want to get rid of a function pointer as it adds a second indirect call to a hot path of the code. I do not change vtables.

User avatar
Yakk
Poster with most posts but no title.
Posts: 11045
Joined: Sat Jan 27, 2007 7:27 pm UTC
Location: E pur si muove

Re: Coding: Fleeting Thoughts

Postby Yakk » Tue Jul 26, 2016 1:13 pm UTC

Ah yes, that is fine.

I'm just saying do:

Code: Select all

template<class T> struct tag_t { constexpr tag_t() {}; };
template<class T> constexpr tag_t<T> tag{};

template<class T, std::size_t Size=sizeof(T), std::size_t Align=alignof(T)>
struct poly_auto {
  template<class U>
  U* ctor( U&& u ) {
    static_assert( sizeof(U) < Size );
    static_assert( alignof(U) < Align );
    static_assert( std::is_base_of<T, U>{}, "U must be derived from T" );
    auto* r = new(&storage) U(std::move(u));
    ptr = r;
    return r;
  }
  template<class U, class...Args,
    std::enable_if_t<std::is_constructible<U, Args...>{}, int>* = nullptr
  >
  U* ctor( Args&&...args ) {
    static_assert( sizeof(U) < Size );
    static_assert( alignof(U) < Align );
    static_assert( std::is_base_of<T, U>{}, "U must be derived from T" );
    auto* r = new(&storage) U(std::forward<Args>(args)...);
    ptr = r;
    return r;
  }
  template<class U, class L, class...Args,
    std::enable_if_t<std::is_constructible<U, std::initializer_list<L>&, Args...>{}, int>* = nullptr
  >
  U* ctor( std::initializer_list<L> il, Args&&...args ) {
    static_assert( sizeof(U) < Size );
    static_assert( alignof(U) < Align );
    static_assert( std::is_base_of<T, U>{}, "U must be derived from T" );
    auto* r = new(&storage) U(il, std::forward<Args>(args)...);
    ptr = r;
    return r;
  }
  template<class U, class...Args>
  poly_auto( tag_t<U>, Args&&...args ) {
    ctor<U>(std::forward<Args>(args)...);
  }
  template<class U, class L, class...Args>
  poly_auto( tag_t<U>, std::initializer_list<L> il, Args&&...args ) {
    ctor<U>(il, std::forward<Args>(args)...);
  }
 
  T* get() { return ptr; }
  T const* get() const { return ptr; }
 
  poly_auto(poly_auto const&)=delete; // if we want this, we need to store how to do it
  poly_auto(poly_auto &&)=delete; // if we want this, we need to store how to do it
private:
  T* ptr = nullptr;
  std::aligned_storage_t<Size, Align> storage;
};

which wraps up the storage stuff with a safely-derived-pointer-to-T.
One of the painful things about our time is that those who feel certainty are stupid, and those with any imagination and understanding are filled with doubt and indecision - BR

Last edited by JHVH on Fri Oct 23, 4004 BCE 6:17 pm, edited 6 times in total.

User avatar
Xeio
Friends, Faidites, Countrymen
Posts: 5086
Joined: Wed Jul 25, 2007 11:12 am UTC
Location: C:\Users\Xeio\
Contact:

Re: Coding: Fleeting Thoughts

Postby Xeio » Thu Jul 28, 2016 7:44 pm UTC

It's so satisfying to learn that subtracting two nullable DateTime objects results in a nullable Timespan.

Not that it should be be surprising, I just hadn't tried it before.

User avatar
Robert'); DROP TABLE *;
Posts: 730
Joined: Mon Sep 08, 2008 6:46 pm UTC
Location: in ur fieldz

Re: Coding: Fleeting Thoughts

Postby Robert'); DROP TABLE *; » Thu Jul 28, 2016 11:42 pm UTC

FT: I need to pay more attention when the compiler warns, "Implicit declaration of function." Especially when that function returns something other than int32.
...And that is how we know the Earth to be banana-shaped.

User avatar
Xeio
Friends, Faidites, Countrymen
Posts: 5086
Joined: Wed Jul 25, 2007 11:12 am UTC
Location: C:\Users\Xeio\
Contact:

Re: Coding: Fleeting Thoughts

Postby Xeio » Fri Jul 29, 2016 3:43 am UTC

The Windows.Media.OCR library is pretty nice and works better than Tesseract... except it's locked to UWP...

And UWP can't do screen capturing, so, *sigh*.

User avatar
Link
Posts: 1327
Joined: Sat Mar 07, 2009 11:33 am UTC
Location: ᘝᓄᘈᖉᐣ
Contact:

Re: Coding: Fleeting Thoughts

Postby Link » Sat Jul 30, 2016 8:03 am UTC

korona wrote:I'm not relying on undefined behavior. I'm relying on behavior that is defined in the ABI and not in the C++ standard. There is a huge difference between undefined and implementation defined behavior. If you ever called a POSIX or WinAPI function you're doing exactly the same.

Alright, I suppose that's fair, if it's unambiguously guaranteed by the ABI and you can be certain that the majority of your users have a compliant system.

User avatar
Xeio
Friends, Faidites, Countrymen
Posts: 5086
Joined: Wed Jul 25, 2007 11:12 am UTC
Location: C:\Users\Xeio\
Contact:

Re: Coding: Fleeting Thoughts

Postby Xeio » Sat Jul 30, 2016 4:18 pm UTC

Xeio wrote:
The Windows.Media.OCR library is pretty nice and works better than Tesseract... except it's locked to UWP...
Scratch that, I got it working outside of UWP...

Now I have to weigh the fact that what I have done is probably terrible and I should feel bad...

On the other hand... it works.

EDIT: If you're wondering why something is so slow and such a memory/CPU hog, make sure your timer isn't running in ticks, rather that milliseconds.

Turns out running something 10,000 times more often that you were trying to is... slowish.

MuEta
Posts: 15
Joined: Tue Dec 22, 2015 2:44 pm UTC

Re: Coding: Fleeting Thoughts

Postby MuEta » Tue Aug 02, 2016 12:44 pm UTC

Xeio wrote:
Xeio wrote:
The Windows.Media.OCR library is pretty nice and works better than Tesseract... except it's locked to UWP...
Scratch that, I got it working outside of UWP...

Now I have to weigh the fact that what I have done is probably terrible and I should feel bad...

On the other hand... it works.

EDIT: If you're wondering why something is so slow and such a memory/CPU hog, make sure your timer isn't running in ticks, rather that milliseconds.

Turns out running something 10,000 times more often that you were trying to is... slowish.


Uhh... Any chance you could share your code/implementation to get that to work (or make a blog post, which would be awesome)? I'm working with tesseract right now and want to try out my other options.

User avatar
Xeio
Friends, Faidites, Countrymen
Posts: 5086
Joined: Wed Jul 25, 2007 11:12 am UTC
Location: C:\Users\Xeio\
Contact:

Re: Coding: Fleeting Thoughts

Postby Xeio » Tue Aug 02, 2016 2:42 pm UTC

MuEta wrote:Uhh... Any chance you could share your code/implementation to get that to work (or make a blog post, which would be awesome)? I'm working with tesseract right now and want to try out my other options.
Even better, repository, though maybe a writeup would be a good idea... :mrgreen:

https://github.com/Xeio/VoidRewardParser

The magic is mostly just adding a reference to "C:\Program Files (x86)\Windows Kits\10\UnionMetadata\Windows.winmd". OCR code is in ScreenCapture.cs.

You may also need the WindowsRuntime.dll reference from .Net Core... I don't recall exactly what I added that for, it may have been the stream stuff (the framework doesn't make it particularly easy for a MemoryStream to talk to a InMemoryRandomAccessStream).

Be warned that I'm like 99% sure this will only work on Windows 10 if you do that though unless you figure out a way to package it (I don't have a Win 7 or earlier machine to test on, but it's possible doing something like including the winmd file would work, I have no idea).

MuEta
Posts: 15
Joined: Tue Dec 22, 2015 2:44 pm UTC

Re: Coding: Fleeting Thoughts

Postby MuEta » Tue Aug 02, 2016 5:27 pm UTC

Xeio wrote:Even better, repository, though maybe a writeup would be a good idea... :mrgreen:


I haven't tried it yet, but you might just be my favorite person for today.

Also, remember kids, read the fucking manual. Sometimes, StackOverflow says that a value will be in [1] instead of [0], and you will cry yourself to sleep because your output is nonsense, but when you actually do your due diligence, you might save yourself from crippling depression and a receding hairline.

ThemePark
Posts: 450
Joined: Fri Jun 27, 2008 5:42 pm UTC
Location: Århus, Denmark

Re: Coding: Fleeting Thoughts

Postby ThemePark » Tue Aug 02, 2016 9:18 pm UTC

Xeio wrote:
Xeio wrote:
The Windows.Media.OCR library is pretty nice and works better than Tesseract... except it's locked to UWP...
Scratch that, I got it working outside of UWP...

Now I have to weigh the fact that what I have done is probably terrible and I should feel bad...

On the other hand... it works.

To paraphrase a famous quote: If it's fixed, don't break it. As long as it's just for private use, if it works that's what matters.
I have traveled from 1979 to be a member of the unofficial board Council of Elders. Phear M3

User avatar
Xeio
Friends, Faidites, Countrymen
Posts: 5086
Joined: Wed Jul 25, 2007 11:12 am UTC
Location: C:\Users\Xeio\
Contact:

Re: Coding: Fleeting Thoughts

Postby Xeio » Wed Aug 03, 2016 7:32 pm UTC

MuEta wrote:I haven't tried it yet, but you might just be my favorite person for today.
If you were fiddling with Tesseract, did you ever run into it being weird about layout, and a way to fix that?

I was trying to at least get some sort of compatibility on Win 7 working, but I can't find any way for it to automatically detect the layout better. Microsoft's OCR library just seems to be "smart" by default, and detects text of different sizes and separated by space as different blocks... I'm just not sure if Tesseract is just more focused on text than layout.

User avatar
Yakk
Poster with most posts but no title.
Posts: 11045
Joined: Sat Jan 27, 2007 7:27 pm UTC
Location: E pur si muove

Re: Coding: Fleeting Thoughts

Postby Yakk » Wed Aug 10, 2016 5:12 am UTC

C++17 adds any (anything copyable) and variant (tagged union).

I like type erasure. And I like operator->*

So my latest insanity is:

Code: Select all

constexpr any_method<void(std::ostream&)> println = [](auto&& self, auto&& os){ os << self << '\n'; };

std::variant<int, double> v = 3.14;
(v->*println)(std::cout);

super_any<&println> a = "hello world"s;

(a->*println)(std::cout);

(7->*println)(std::cout);

(std::make_unique<double>(-42)->*println)(std::cout);

Modular type-erasing pseudo-method-pointers.

Heh.
One of the painful things about our time is that those who feel certainty are stupid, and those with any imagination and understanding are filled with doubt and indecision - BR

Last edited by JHVH on Fri Oct 23, 4004 BCE 6:17 pm, edited 6 times in total.

EvanED
Posts: 4324
Joined: Mon Aug 07, 2006 6:28 am UTC
Location: Madison, WI
Contact:

Re: Coding: Fleeting Thoughts

Postby EvanED » Wed Aug 10, 2016 5:32 am UTC

I think I might have to work on my program's memory use.

353gb.png
353gb.png (11.1 KiB) Viewed 3780 times


(You might think that this would be thrashing like crazy, but it was actually pretty well-behaved most of the time, as you can see from CPU% there. Though it is thrashing now as it's running destructors after program exit. I did have to go and make 200 GB of swap to add; I was worried it'd run out. I actually have very little idea of what's taking up the space. I'm hoping I'm just a moron and it's something really stupid.)

commodorejohn
Posts: 958
Joined: Thu Dec 10, 2009 6:21 pm UTC
Location: Placerville, CA
Contact:

Re: Coding: Fleeting Thoughts

Postby commodorejohn » Wed Aug 10, 2016 6:20 am UTC

Man, JQuery is really getting out of hand these days.
"'Legacy code' often differs from its suggested alternative by actually working and scaling."
- Bjarne Stroustrup
www.commodorejohn.com - in case you were wondering, which you probably weren't.

User avatar
Xanthir
My HERO!!!
Posts: 5214
Joined: Tue Feb 20, 2007 12:49 am UTC
Location: The Googleplex
Contact:

Re: Coding: Fleeting Thoughts

Postby Xanthir » Wed Aug 10, 2016 4:36 pm UTC

How so? I haven't followed it in years.
(defun fibs (n &optional (a 1) (b 1)) (take n (unfold '+ a b)))

commodorejohn
Posts: 958
Joined: Thu Dec 10, 2009 6:21 pm UTC
Location: Placerville, CA
Contact:

Re: Coding: Fleeting Thoughts

Postby commodorejohn » Wed Aug 10, 2016 4:48 pm UTC

(It was a joke on EvanED's post. Though really, I've encountered web applications where I really wouldn't have been surprised to find out they were using a significant fraction of a terabyte of swap...)
"'Legacy code' often differs from its suggested alternative by actually working and scaling."
- Bjarne Stroustrup
www.commodorejohn.com - in case you were wondering, which you probably weren't.

User avatar
Flumble
Yes Man
Posts: 1944
Joined: Sun Aug 05, 2012 9:35 pm UTC

Re: Coding: Fleeting Thoughts

Postby Flumble » Wed Aug 10, 2016 5:45 pm UTC

commodorejohn wrote:(It was a joke on EvanED's post. Though really, I've encountered web applications where I really wouldn't have been surprised to find out they were using a significant fraction of a terabyte of swap...)

Are they even capable of allocating more than 4GiB these days? :o

My firefox gets crashy after allocating a mere 1.3GB (=the number that task manager reports)... well, that and long-running (in the order of weeks) incognito sessions. Well, it usually doesn't crash but just gets less responsive and starts to continuously spend time garbage-collecting.

User avatar
ucim
Posts: 5564
Joined: Fri Sep 28, 2012 3:23 pm UTC
Location: The One True Thread

Re: Coding: Fleeting Thoughts

Postby ucim » Wed Aug 10, 2016 6:37 pm UTC

Flumble wrote:and starts to continuously spend time garbage-collecting.
Isn't that what incognito sessions are for?

Jose
Order of the Sillies, Honoris Causam - bestowed by charlie_grumbles on NP 859 * OTTscar winner: Wordsmith - bestowed by yappobiscuts and the OTT on NP 1832 * Ecclesiastical Calendar of the Order of the Holy Contradiction * Please help addams if you can. She needs all of us.

commodorejohn
Posts: 958
Joined: Thu Dec 10, 2009 6:21 pm UTC
Location: Placerville, CA
Contact:

Re: Coding: Fleeting Thoughts

Postby commodorejohn » Wed Aug 10, 2016 6:46 pm UTC

Flumble wrote:
commodorejohn wrote:(It was a joke on EvanED's post. Though really, I've encountered web applications where I really wouldn't have been surprised to find out they were using a significant fraction of a terabyte of swap...)

Are they even capable of allocating more than 4GiB these days? :o

I dunno, but it's not for lack of trying, lemme tell ya...
"'Legacy code' often differs from its suggested alternative by actually working and scaling."
- Bjarne Stroustrup
www.commodorejohn.com - in case you were wondering, which you probably weren't.

User avatar
Flumble
Yes Man
Posts: 1944
Joined: Sun Aug 05, 2012 9:35 pm UTC

Re: Coding: Fleeting Thoughts

Postby Flumble » Wed Aug 10, 2016 9:07 pm UTC

ucim wrote:
Flumble wrote:and starts to continuously spend time garbage-collecting.
Isn't that what incognito sessions are for?

Are you implying I have a bad taste in porn?! :o

No but even without interacting with the browser, an old (incognito) session takes up 30% of one core continuously and keeps allocating and deallocating tens of MBs. I know it's conflicting to "not save anything from this browser session" and "remember all this while the session is open", but firefox shouldn't get all schizophrenic because of it.

User avatar
phlip
Restorer of Worlds
Posts: 7543
Joined: Sat Sep 23, 2006 3:56 am UTC
Location: Australia
Contact:

Re: Coding: Fleeting Thoughts

Postby phlip » Wed Aug 10, 2016 11:34 pm UTC

Flumble wrote:
Are you implying I have a bad taste in porn?! :o

I'm more intrigued by your porn-viewing sessions that last in the order of weeks...

Code: Select all

enum ಠ_ಠ {°□°╰=1, °Д°╰, ಠ益ಠ╰};
void ┻━┻︵​╰(ಠ_ಠ ⚠) {exit((int)⚠);}
[he/him/his]

EvanED
Posts: 4324
Joined: Mon Aug 07, 2006 6:28 am UTC
Location: Madison, WI
Contact:

Re: Coding: Fleeting Thoughts

Postby EvanED » Thu Aug 11, 2016 3:05 am UTC

commodorejohn wrote:(It was a joke on EvanED's post.
If it makes you feel better, I knew what you were doing. :-)

commodorejohn
Posts: 958
Joined: Thu Dec 10, 2009 6:21 pm UTC
Location: Placerville, CA
Contact:

Re: Coding: Fleeting Thoughts

Postby commodorejohn » Thu Aug 11, 2016 7:04 am UTC

It helps :D
"'Legacy code' often differs from its suggested alternative by actually working and scaling."
- Bjarne Stroustrup
www.commodorejohn.com - in case you were wondering, which you probably weren't.

User avatar
Xeio
Friends, Faidites, Countrymen
Posts: 5086
Joined: Wed Jul 25, 2007 11:12 am UTC
Location: C:\Users\Xeio\
Contact:

Re: Coding: Fleeting Thoughts

Postby Xeio » Sun Aug 14, 2016 5:46 pm UTC

It's be nice if Git worked with Unicode text files...

I don't even know why, but I merged someone's commit, and now git continually detects the file as "changed" even if I delete and recreate it from the current commit.

And of course git can't do a compare (either on GitHub or locally) because it's a "binary" file since it's Unicode. I have no idea why it even detects it as a different file.

Ubik
Posts: 1016
Joined: Thu Oct 18, 2007 3:43 pm UTC

Re: Coding: Fleeting Thoughts

Postby Ubik » Sun Aug 14, 2016 6:37 pm UTC

I suppose it's an UTF-16 file (or less likely, UTF-32), as cursory googling shows Git doesn't process wide encodings as text. It shouldn't be about Unicode itself - if the file was encoded as UTF-8, my guess is that it would work then...assuming there still aren't actual null characters or similar lurking in the file, that keep making it look like binary to Git.

That the file keeps on showing as modified sounds suspicious.

Tub
Posts: 310
Joined: Wed Jul 27, 2011 3:13 pm UTC

Re: Coding: Fleeting Thoughts

Postby Tub » Sun Aug 14, 2016 8:09 pm UTC

I'm pretty sure that multiple projects full of utf8 encoded text files work perfectly fine with git on any operating system I've used.

It is possible to run into trouble if your operating system cannot create an exact copy of the file, e.g. files larger than 2 or 4 GB, or files with reserved names (like aux, nil, prt, lpt and others on windows, which includes variants like aux.cpp), or FOO and foo on a case-insensitive filesystem, or git repositories with permissions that your filesystem does not support, or if you're going through a network share with similar restrictions. When the operating system does not return exactly the file it was given, git has no choice but to assume it was changed.

User avatar
Xeio
Friends, Faidites, Countrymen
Posts: 5086
Joined: Wed Jul 25, 2007 11:12 am UTC
Location: C:\Users\Xeio\
Contact:

Re: Coding: Fleeting Thoughts

Postby Xeio » Sun Aug 14, 2016 8:27 pm UTC

Ubik wrote:That the file keeps on showing as modified sounds suspicious.
No idea, git seems to detect a change in the file. I might have to make a dummy commit just so it goes away.

Tub wrote:It is possible to run into trouble if your operating system cannot create an exact copy of the file, e.g. files larger than 2 or 4 GB, or files with reserved names (like aux, nil, prt, lpt and others on windows, which includes variants like aux.cpp), or FOO and foo on a case-insensitive filesystem, or git repositories with permissions that your filesystem does not support, or if you're going through a network share with similar restrictions. When the operating system does not return exactly the file it was given, git has no choice but to assume it was changed.
Nope, nothing fancy, it's a tiny file on the system drive.

It's not even a new file to the merge, they changed a line.

For added hilarity, I went to the command line:

Image

So much for git reset.


Return to “Coding”

Who is online

Users browsing this forum: No registered users and 5 guests