New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Interoperability issues with dependencies using Objective-C in Cluster Mode on macOS 10.13 High Sierra #1421
Comments
Relevant Unicorn thread https://bogomips.org/unicorn-public/20170804191023.GA28511@dcvr/T/ |
I'm with Eric. I have no idea how ObjC factors into this. |
ObjC factors in because if anything within the forked process links against ObjC APIs, it will be unusable from within Puma! :) I’m doing some more debugging to determine what exactly is causing this. |
Oh crap. Subscribe me, github! |
Having trouble doing more debugging, as the child exits as soon as it encounters this error, so attaching rbtrace seems difficult if not impossible. |
Sorry to butt in, but do you know if this effects I can't install High Sierra since many of the professional applications I use daily don't support it just yet (I'm also a musician). |
Anything that forks and runs user-specified code within that fork without using It is unclear exactly where a solution will need to be implemented. |
I’ve managed to track down what appears to be the source of this error in the application I’m working with, it’s loading It seems unlikely that we can get away with our workers not being able to talk to our database on macOS… 🤔 Again, this is not that you have to actively be using any features provided by these Objective-C frameworks, their inclusion in a thing you link against is enough to trigger this error. We do not use the Kerberos nor LDAP features of Postgres in our application, and yet here we have this error. |
Apparently this thread has ended up on reddit where it seems to be being misunderstood. Regarding this comment:
The issue is that code within Ruby can call fork (as Puma does), and loading dynamic libraries from Ruby or a Ruby native extension can cause an Objective-C class to be |
@ticky , I'm happy you're managing to track the issue down and I'm thankful for the added information. However, reading the information it wasn't clear to me if
Just my 2¢, I don't know if any of it helps. P.S. You posted the issue summery as I was writing this and I wondered if, perhaps, some non Objective-C function calls are implemented using the Objective-C framework under the hood..? |
I don’t believe
This is the best hypothesis I now have. It looks like it might work if those things are all loaded before forking, however, I am not familiar enough with the way our system is set up to know how the should work!
I don’t believe it’s that they are intentionally calling out to any particular Obj-C method, but merely linking to their Obj-C frameworks which requires the Obj-C initialisation to occur! |
|
I had tried to activate single-process mode earlier but not had any luck. Turns out you need to remove the Running in single mode (confirmed with Thus, it’s the |
A bit of an update - this behaviour also doesn’t cause a crash if |
I have a Unicorn app, rather than a Puma app, but something that's working for me is to explicitly require a gem with ObjC linkage within a |
@mistydemeo’s onto something! I just tried adding I wonder if there’s a less strange-looking way to achieve this than loading a specific dependency we know to link ObjC. 🤔 |
Okay, this doesn’t look much less strange but it is, at least, only relying on side effects it’s explicitly calling upon! # Work around macOS 10.13 and later being very picky about
# `fork` usage and interactions with Objective-C code
# see: <https://github.com/puma/puma/issues/1421>
if /darwin/ =~ RUBY_PLATFORM
before_fork do
require 'fiddle'
# Dynamically load Foundation.framework, ~implicitly~ initialising
# the Objective-C runtime before any forking happens in Puma
Fiddle.dlopen '/System/Library/Frameworks/Foundation.framework/Foundation'
end
end As Misty noted, this ensures the the Objective-C runtime has been initialised (a side effect instigated by the dynamic loader itself) before any Whether this is a workaround which makes sense to include in Puma is definitely debatable, but it appears that many things which make use of this sort of fork model will need to consider it, and perhaps also document it. Note also that I did not guard this to only occur on 10.13, just macOS in general, as this should work fine on other versions, and in fact be more “correct” behaviour by its definition. |
Updated the initial post with the updated details of this issue! |
Great work @ticky! I won't be able to take a closer look at this until next week - holding off on my own High Sierra upgrade until I am back home and can make some backups first. |
According to this article, setting |
@tenderlove yep, I noted that in my initial post |
It would be possible to have a quite exception instead of the Maybe something like: # Work around macOS 10.13 and later being very picky about
# `fork` usage and interactions with Objective-C code
# see: <https://github.com/puma/puma/issues/1421>
if /darwin/ =~ RUBY_PLATFORM
begin
require 'fiddle'
# Dynamically load Foundation.framework, ~implicitly~ initialising
# the Objective-C runtime before any forking happens in Puma
Fiddle.dlopen '/System/Library/Frameworks/Foundation.framework/Foundation'
rescue Exception => _e
end
end
Since However, the |
@boazsegev my last message was regarding the use of the environment variable ( As @mistydemeo suggested, it works in Unicorn too, and it likely works in any other thing which is using fork in a similar way, and running arbitrary Ruby code within. I would suggest that having an option to implicitly load Foundation would make some sense, however, I would be reluctant to suggest having it always implicitly do this for all users, as it feels very heavy-handed. It’s unfortunate that we can’t catch these kinds of errors in our code (the process is killed instantly), or I’d suggest adding our own message about it! It’s also worth noting again that this behaviour doesn’t trigger this error case outside of cluster mode, or if the application is preloaded. |
@ticky , you're probably right about it being heavy-handed, but...
I don't think it does. Especially since Other C extensions (such as I might be over thinking this, since the issue was opened here, but the discussion feels as if it's more than about a Puma specific solution (it includes Unicorn, Sidekiq etc'). Just my 2¢, of course. |
I don’t mean the You are right, though, that the issue is bigger than Puma. It’s likely that Puma, Unicorn, Passenger and Iodine will all need to consider the implications of this macOS change. Unicorn’s maintainer, though, seems to have already expressed a disinterest in doing so! Either that or it requires some sort of structural change in Ruby (though that seems rather unlikely to fly) |
Apple added an assertion in 10.13 that prevents anything Obj-C related from occuring between fork and exec. This workaround prevents the assertion. See: - http://www.sealiesoftware.com/blog/archive/2017/6/5/Objective-C_and_fork_in_macOS_1013.html - puma/puma#1421
It looks like the dlopen workaround is the best solution in terms of fewest side-effects. Regarding @ticky's comment about structural changes in Ruby: I think this goes beyond Ruby itself. Forking is fundamentally problematic when there is multithreading involved. We can make it work, and in the context of Ruby app servers it usually works well so far, but that's in spite of a lack of real solutions. To guarantee that forking is safe, the application must not be running any threads at the point of fork (or the other threads must only be executing async-signal-safe code). MRI can guarantee this about its own code to some degree, but third party native libraries may spawn their own threads. So any gems that the user pulls in should not spawn threads until the app server has forked. Gems with multithreaded native extensions should provide an explicit thread initialization method, which the app only calls after the app server has forked. This requires cooperation in the entire gem ecosystem. Does something like the dlopen workaround belong in Puma/Unicorn/iodine/Passenger? This is more of a philosophical question than a technical one. I cannot answer for Puma/Unicorn/iodine, but Passenger's philosophy is put user experience at the forefront, so we have chosen to include the dlopen workaround in the next Passenger version. |
Unless it has other side effects, I’d suggest implementing this within Puma, Unicorn and iodine would cause the fewest headaches for implementors and users. Barring some incompatibility we haven’t yet encountered with initialising the Objective-C runtime within our Ruby process, I think it makes the most sense! |
IMHO, since the issue effects High Sierra, I believe it is acceptable to expect users to upgrade to a High Sierra compatible Ruby version. I also think that back porting the patch is wise. |
Yeah, I agree with what @ticky and @boazsegev said. Also, since there is an environment variable setting which gives you "the old behavior", it's easier for us to just point users to that instead. IMO "upgrade your ruby patchlevel" or "use this environment variable to go back to the way things were pre-2017" is an OK solution. |
UPDATE: It seems a patch was merged with the Ruby trunk. This probably means the issue will be resolved in future Ruby releases (~~~the next 2.4.x version should be patched~~~ EDIT: nope, still waiting on a patch, maybe 2.5?). |
Okay, because of the quick fix here I'm going to close this from our perspective. This will be fixed in Ruby 2.4.3 (which will probably be around Christmas), as of this writing backports to other Ruby minor versions are unknown. To work around this issue today, I suggest just setting the |
As an update from the iodine server perspective: The iodine server now includes a fix for this issue (starting at version 0.4.10), allowing people using The decision was made for two reasons:
I'm not sure everyone needs to do the same, but I thought it might be of interest. |
I’d warn that as discussed above, this potentially masks other issues which this new behaviour is attempting to mask. Loading Foundation explicitly will avoid the “implicit Obj-C runtime initialisation in a fork” errors, but not mask similar but distinct errors in which your program actually triggers the behaviour this intends to disallow. |
I think you meant "attempting to unmask" right? My view is that Apple basically turned a feature into a bug here and made a helpful safety check cause a crash. Puma already warns when multiple threads exist before forking, so the safety check here is not as helpful as it might otherwise be. If people want the additional safety check Apple wants to provide here, they can upgrade to a newer Ruby version (when it comes out). Until then, they can live with the old behavior via the environment variable workaround. |
Yep :)
The safety check won’t apply to any C extensions which may fork, though, right? |
Correct, we're only checking the Ruby |
@nateberkopec , I'm not sure what the common practice is, but I re-assesed the observation that language extensions (C / Java) could call It seems that Ruby has a "timer thread" running in the background. After reading through the Ruby source code I realized that calling Extensions that call As for the ...however, some extensions might spawn new native threads that never call the Ruby API (iodine does this for IO flushing). These threads are obviously not in the |
@nateberkopec @ticky unfortunately it seems the issue is still present even with Ruby 2.4.3 I just encountered it with Ruby 2.4.3, macOS High Sierra 10.13.2 and puma 3.11.0:
I've isolated it, in my code, to these three lines of code which crash the process with our without preloading: Faraday.new(url: url) do |faraday|
faraday.adapter :patron
end @ticky's patch did the trick |
this still seems to be happening for me in puma's cluster mode, not in single mode.
|
@webark yeah the issue is because of the usage of forking in OSX and how it changed in High Sierra, try @ticky's patch here: #1421 (comment) |
@rhymes, @webark: the upstream Ruby patch hasn’t been backported to Ruby 2.4. It looks like it will in fact land in Ruby 2.5, which is forthcoming. @nateberkopec, it might be worth reconsidering the “no Puma workaround” approach given there will potentially be people wanting to stay on 2.4 and I don’t have confirmation of whether any backport is planned. |
@ticky amazing! thanks for the help and support! |
Update: a backport for 2.4.x has landed, it should be in the 2.4.4 release, though I suspect that won’t appear until the new year! |
Hi everyone! I am seeing the same I first saw the crash when I ran
I tested both
|
I wish I could test this, but I run my music studio on the same computer and can't update to Catalina just yet (not until all the pro-audio software is compatible). However, I thought this was solved in the Ruby core (this should probably be considered a Ruby bug rather than a Puma specific issue). You can see the code that links agains the Objective C Framework in the Ruby repo. It should have solved the issue... unless (maybe) you might be using custom compilation flags or a different compiler. I am curious if a dynamic patch / workaround could be applied, like I did with iodine (only using fiddle). Does the issue occur with iodine? Kindly, |
@ndbroadbent can you open a new issue and describe how to reproduce the crash? (i.e. share code that makes it crash) |
We're crashing again (in macOS) since Homebrew/homebrew-core#132976 This has occured before, from https://bugs.ruby-lang.org/issues/16239 > This is a fatal interaction between the PostgreSQL 12 client libraries > and the GSS implementation provided by macOS. This is being tracked in > the pg gem at ged/ruby-pg#311 More in ged/ruby-pg#311 (comment) Docs on PGGSSENCMODE https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNECT-GSSENCMODE OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES has been a workaround in the past (puma/puma#1421) but it shouldn't be used without care: Homebrew/homebrew-core#137909 (comment)
Steps to reproduce
Install macOS 10.13 High Sierra
Freshly install Ruby and Puma
Execute an application which uses the
pg
gem, does not usepreload_app!
, and uses Puma in cluster modeExpected behavior
Puma should work and execute jobs as expected
Actual behavior
Puma fails, repeatedly outputting these errors from worker processes:
System configuration
Ruby version: 2.4.2
Rails version: 5.1.3
Puma version: 3.10.0
macOS High Sierra changes the behaviour of the
fork
syscall such that calls to Objective-C APIs in forked processes are treated as errors.It appears some part of Puma’s use of Ruby violates this, which means Puma doesn’t work correctly in cluster mode on 10.13.A temporary workaround is to apply theOBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES
environment variable to the parent puma process, which will then trigger the behaviour observable in macOS versions 10.12 and earlier.EDIT: Puma uses
fork
in cluster mode, and no special consideration is given to what runs within the fork. It is possible to depend upon a gem, or dynamically load a shared object, which in turn links to an Objective-C framework. This linkage will implicitly call an Objective-Cinitialize
method as it sets up the Objective-C runtime for the first time, and if this call occurs within a fork, the program will be killed, and the above error message printed.A less heavy-handed workaround is to use a
before_fork
callback like this:This will implicitly initialise the Objective-C framework within the Puma process, allowing forked children to use other Objective-C frameworks.
The text was updated successfully, but these errors were encountered: