First, I think the ultimate goal of strong AI, not weak AI, is to replace humans. Second, if that happens (not in a million years!), that raises serious ethical issues with implications in Theology, anthropology, and ontology, to consider just a few areas.

In theology, questions like these need answers. “Since God created man in His image (Imago Dei), and in theory, AI aims to progress to the level of a human being, would it be made in the image of God?” Or would AI be simply the image of man (imago humanis)? In anthropology, AI will consider redefining what it means to be human. Last but not least, in ontology, can computers be considered as agents? Now, if one says computers can be agents, then, one needs to answer the question: how functional must a machine be to claim human-level consciousness, intelligence, and nature?

What I know for sure is that one may approach AI differently, but what humanity will mostly get from this AI euphoria is the consequence of doing AI with an emphasis on a human-centered approach. As Rich and Knight say, this approach has been followed by people who believe that AI is the study of making computers do things that people are better at doing at the moment. That could be a problematic approach.  In reverse, a rational approach, to me, is not controversial, since in most cases, it is already very beneficial to humanity, considering all the gadgets we use daily. Their relevancy, in that sense, makes AI truly universal.

Thus, the concern is whether we want AI to model or replace humans. Defending the view that computers will reach a point at which they don’t need human beings at all can raise serious ethical issues. Before we know it, people could start treating artificial intelligence as an end in itself, as if one could rely solely on AI and not consider human input at all. Computers cannot be genuinely intelligent or conscious. AI programs can do what programmers tell them to do. After all, human beings remain the makers while these machines are nothing more than human inventions.

Moreover, since we are creative beings made in the image of God (which we truly are), creating machines that think like us or do what we are capable of doing poses, as time will continue to reveal, a serious threat to humanity.