Introduction To Niantic Lightship
Avid gamers can gain weight, develop awkward social habits, and lose touch with the physical world.
Some gamers don’t even know their neighbours.
The US gaming and software development company, Niantic, Inc. set to change
Their mega-popular augmented reality game, Pokémon Go, attracted over 100 million players worldwide.
Not only that — the game has clinically proven physical health benefits. The National Library of Medicine showed that Pokémon Go players increased their physical activities by 25%.
But what’s more impressive?
The NCBI study says, “We find that Pokémon Go is able to reach low activity populations, whereas all 4 leading mobile health apps studied in this work largely draw from an already very active population.”
Niantic established themselves as one of specialized AR developers who cares about holistic gaming experiences that are fun, innovative, and healthy.
Their latest software tool - Niantic Lightship, they aim to leverage developers to express themselves through Niantic’s Lightship platform and build engaging and useful AR applications.
Read on to discover how Niantic enables a developer-led, world’s first real-world metaverse and get a complete starter guide for Lightship developers.
Who is Niantic
John Hanke founded Niantic Labs in 2010 under Google as an internal startup. When Google restructured in 2015 to become a subsidiary of Alphabet, Niantic Labs became an independent entity.
This image is a placeholder for this video: Why Niantic?
Most people would remember the hit AR game Pokémon Go, which has received over 1 billion downloads since its launch and remains one of the most played AR games worldwide.
Niantic or Niantic Labs is the company behind Pokémon Go. But Pokémon Go isn’t Niantic’s only successful game.
The AR games company has enjoyed massive fame for other games like Ingress and Pikmin Bloom.
Ingress has attracted millions of players and has 4,000 dedicated communities. Pikmin Bloom has received over two million downloads, and users have taken over 338 billion steps.
In short, Niantic has the world’s heart as far as AR games are concerned.
Niantic's Big Goal
Niantic’s big goal is to build a real-world metaverse, and they’ve received funding to pursue it.
Goal: Build a Real-World Metaverse
In the last quarter of 2021, Niantic announced its ambitious goal to build a real-world metaverse where virtual objects can interact and harmonize with real-world environments.
Niantic attracts tens of millions of active players every month. Collectively, these players have walked a total of 17.5 billion kilometers or 10.9 billion miles. That’s like walking the earth’s circumference more than 436,600 times.
So, Niantic wants to channel that high user engagement into its most audacious project yet—the real-world metaverse. Niantic has already attracted $300 million in venture funding for this project.
How Niantic is Building Its Real-World Metaverse
Niantic’s strategy for achieving its goals includes these:
- Providing developers with supporting toolkits
- Forging key brand partnerships
- Partnering with studios
Let’s explore them.
Provide Developer Tools and Enable an Active Developer Community
This $9 billion-dollar gaming behemoth aims to achieve its lofty target by empowering developers with toolkits, community, and funding.
Niantic released the Augmented Reality Developer Kit (ARDK) through its developer platform, lightship.dev, and provides supporting developer documentation to help users.
This image is a placeholder for this video: https://youtu.be/wX7sKLPtrsI
Niantic maintains a Showcase page and a Forum to inspire creators and enable instant community help and project feedback from members.
Developers can access Lightship’s core APIs for free regardless of how many app users they acquire.
However, some features have usage limits. For example, apps with less than 50,000 monthly active users can run the Multiplayer API for free, and apps with more users must pay a fee.
Also, developers can use Multiplayer APIs for free for their first six months regardless of their user base size.
In the second quarter of 2022, the company held a developer conference in San Francisco to help its developer community better understand its vision and tools. In addition, Niantic maintains lots of community engagement programs throughout the year.
Niantic has struck key brand and marketing partnerships with Coachella, SoftBank, Universal Picture, Warner Music Group, TRIPP, and many others.
The company has also committed $20 million to support innovative AR projects outside its immediate ecosystem.
Partnering with Studios
Niantic has built studio partnerships with Curiosity, Designium, Heavy.io, Nexus Studios, Preloaded, Trigger Global, and T&S.
Some of Niantic’s key projects with these studios include the Science Museum Group’s AR project and Historic Royal Palaces’ Tower of London moat project with Preloaded. And the T&S project with Shueisha’s Manga Characters.
What is Niantic Lightship
Niantic Lightship is a developer toolset for building augmented reality software tools and apps to enable Niantic’s real-world metaverse.
Lightship’s Augmented Reality Developer Kit (ARDK) provides these solutions:
- Environment-sensing capability, also known as Semantic Segmentation,
- Visual positioning system (VPS) for world-mapping mapping, and
- Enables shared player experiences
Niantic’s products rely on the Lightship Platform, where the company has been building and running popular games like Pikmin Bloom, Pokémon GO, and Ingress.
The platform empowers developers with the resources to create unique augmented reality experiences by harnessing all or some ARDK tools. Niantic has promised to continue rolling out new developer features and expanding the platform’s capabilities.
Top Niantic Lightship ARDK Features
Niantic’s Lightship platform enables developers and players to access and benefit from the core features that power AR experiences, including these:
- Multiplayer interactions for shared experiences
- Semantic Segmentation to understand the physical environment
- Meshing for real-time mapping of unmapped environments
- Visual positioning system (VPS) for placing virtual objects in real-world spaces
Niantic applications run on billions of Android and iOS devices. Its AR technology can work on devices that run on standard RGB sensor cameras and those equipped with the newer LiDAR technology.
Let’s explore each of these features.
Real-Time Physical World Mapping: Meshing
This image is a placeholder for this video: https://youtu.be/wWHAr_TSDWk
Niantic Lightship makes it possible to map any physical space in real-time, especially objects that have not been previously mapped.
The ARDK’s Meshing APIs enable developers to leverage the technological capabilities in smartphone camera sensors to do these:
- Build real-time mesh maps
- Understand the surrounding surfaces and topography of a user
Meshes are responsive to the user’s movements and remain active throughout an app session.
Occlusion APIs give appropriate depths to virtual objects that appear relative to real-world objects. So, for example, if you throw a virtual frisbee over a real-world wall, you’ll continue seeing the wall but not the frisbee.
Real-World Surface Interaction Through Semantic Segmentation
This image is a placeholder for this video: https://youtu.be/1K3DMhOIAJo
Semantic Segmentation is a computer visioning model that identifies and differentiates real-world components like structures, mountains, sky, ground, foliage, water, and more.
Objects interact appropriately with different real-world surfaces using the ARDK’s Semantic Segmentation APIs. Real-world elements inform how virtual content interacts with real space.
Multiplayer Experiences Supported
This image is a placeholder for this video: https://youtu.be/tzbvhxa4ESY
ARDK developers can create ARSessions that support up to five players at a time. The multiplayer API enables users to enjoy simultaneous experiences that feel real in the physical world.
This feature empowers users to enjoy realistic and coherent interactions in virtual spaces that enable real-world collaborations in real-time.
Niantic Lightship’s ARDK keeps the multiplayer interactions fast by using a peer-to-peer networking stack and managed server. As a result, Lightship developers wouldn’t need to write custom codes to support multiplayer events.
In addition, this developer kit avails lightweight multiplayer gaming functions like a synchronized clock, a player lobby system, and session-persistent storage.
Visual Positioning System (VPS)
This image is a placeholder for this video: Lightship Summit | Opening Keynote
The Niantic Visual Positioning System (VPS) helps users enjoy real-world, location-based experiences. So real-world spaces can become meeting places for people to gather, explore, and connect in real-time.
This technology is one of the gaming giant’s most innovative ideas yet. Niantic has always wanted to map the world out in 3D. However, creating that 3D map was their most daunting challenge.
But if Niantic must bring real-world interactivity and information into games, they must solve this piece of the AR puzzle.
The VPS enables players to place virtual objects in real-world spaces for other players using the same application to find. Niantic has mapped thousands of real-world locations to enable this feature to work seamlessly.
Real Use Cases of Niantic Lightship
Niantic Lightship has applications across science, gaming, events, health, and more. Let’s explore some of them.
Historic Royal Palaces Transforms Tower of London Moat
Historic Royal Palaces is an independent charity organization the British government established in 1989 to care for five palaces. It then added a sixth one—Northern Ireland’s Hillsborough Castle in 2014.
This charity organization decided to use AR to reach new audiences everywhere by making historic sites more playful, appealing, and attractive.
In 2022, Historic Royal Palaces teamed up with London’s Preloaded developer studio to change the landscape at the moat around the Tower of London to give an AR flower experience.
Coachella Finally Lands Its Beloved Butterfly on the Spectra Tower
The Spectra tower is a seven-story rainbow walkway tower that launched in Coachella 2018. In the event’s 2022 edition, the mega event introduced the Coachellaverse, an immersive AR experience for program attendees.
Lightship used Semantic Segmentation to help Coachella create an experience with its app for attendees to see a beautiful large-scale butterfly act as if it’s a real-world object.
Science Museum Makes Science for Engaging and Interactive
The Science Museum Group started in the Great Exhibition of 1851. It’s the UK schools’ most visited museum.
This leading science museum group has various collections across technology, mathematics, science, medicine, and engineering.
The Science Museum Group joined forces with the Niantic Lightship to make science more engaging and interactive beyond its walls. So this museum group would use an AR-powered app to connect audiences to the science surrounding them.
Shueisha Brings Manga Characters to the Real World
Shueisha has been publishing comic books since 1926. The Japanese Manga publisher has seen its works developed into animated movies and TV series.
This animation and publishing powerhouse plans to launch an AR-enabled app in 2022 to give life to its Manga characters, including characters from One Piece. They are working with T&S developer studio to get the company’s most loved Manga characters into an AR space.
TRIPP Fosters a Wellness Community Through AR
TRIPP is an extended reality wellness application that offers XR and mobile experiences to improve users’ mind-body health. This app promises to help users do these:
- Shift their moods
- Benefit from binaural audios
- Use guided meditations
- Apply mindfulness methods
- Do breathing exercises
This wellness platform connects its user community through Lightship’s technology. It provides a unique experience that brings users close to a real-world metaverse.
TRIPP says its users have reported mood improvements in as little as eight minutes.
The PGA of America Launches Massive Multiplayer AR Experiences
Niantic Lightship is an outdoors-native, AR-experiences developer tool—it works best in outdoor spaces by design.
Trigger and Niantic worked with the PGA to launch an AR experience, the PGA Jr League Safari Par-Tee mobile app, in the second quarter of 2022.
This interactive, family-friendly mobile game lets users play AR-enabled, collaborative golfing games. The game overlays real-world environments with animal avatars on real-time 3D (RT3D) Adventure Park Golf Courses.
How to Setup and Get Started with Niantic Lightship Development
For a tool that bears so many lofty promises, it’s surprisingly simple to set up. Read on to learn how to get your Lightship developer account, connect it with Unity, and start building.
How to Set Up a Developer Account
Verify that your computer matches the system requirements to develop and run AR experiences with the Lightship ARDK.
You can develop your AR experiences with ARDK in Unity, Android, and iOS. However, you can develop experiences with limited capabilities in Windows and M1 Macs.
Once you’ve ensured your system meets the device, development, and runtime demands for creating experiences with the ARDK, get your Lightship account up and running.
Head over to Lightship and sign up there.
When you’re done signing up, you’ll have the option to explore Niantic Lightship’s community or download the Wayfarer app.
Keep in mind that the Wayfarer app is only available on iOS.
The community showcase lets you see what developers are building.
You’ll find games grouped into these categories:
- Real-Time Mapping
- Semantic Segmentation
Head to your inbox to find your verification email.
Then click the red link to verify your email.
Confirm your email and log in to your account.
The dashboard page would look like this:
Create a New Project and API Token in Your Lightship Account
You can create new projects in two ways once you’ve logged into your account.
If you're adding your first project, you can open a new project from the dashboard.
Just click the Create A Project hyperlink.
But you can always add new projects from the Projects navigation tab.
Click the Plus icon to add a new project.
Click Create A Project.
Add project title and description, and create an API key to connect your project to your Unity editor.
I created a project. So here’s what yours would look like:
You can delete projects too.
Go to Projects and Click Settings. You’ll find the Delete button there. Click on it.
Set Up the Augmented Reality Developer Kit (ARDK) in Unity
The ARDK enables you to create augmented reality (AR) content that melds the real and virtual worlds. For example, you can create virtual objects that hide behind real objects like trees or buildings.
Click the Download tab and scroll down to see sample projects to help you familiarize yourself with what the AR Developer Kit (ARDK) can do.
These examples include:
- Mock meshes
- Mock environments
So you can download these sample projects and upload them to your Unity account.
Lightship also provides you its brand assets, including designs, trademarks, and logos.
So, how do you set up your ARDK in Unity?
Follow these steps:
Step #1: Download the ARDK Kit
Click the Download navigation button to find the latest ARDK version. Then click the download button to get it.
Step #2: Create a Project in Unity
Open your Unity Hub and click the New Project button to add a new project.
Choose a 3D project template, like the 3D Mobile template, and click the Download Template button.
When it’s done downloading the template, name the project and then click Create Project.
Step #3: Create a New or Use an Existing Scene
Use the default sample scene in your new project or create a new scene. You can also import the scene you downloaded from your Lightship account.
Go to Assets, right-click on Scenes > Import Package > Custom Package. Find the folder where you saved the sample assets and click on it to import it into your Unity project.
When the asset file loads, click on the import button.
Step #4: Import the ARDK to Unity as a Custom Package
Open the file selection window in Unity. In your projects, right-click on Assets and select Import Package > Custom Package.
Select your ARDK kit from the folder where you saved it.
Click Import to bring the ARDK into your Unity.
You can upgrade to the latest ARDK version if your project runs on an older version. Do these:
- Copy and save any additions or changes you’ve made to the existing ARDK package’s Assets/ARDK or Assets/ARDKExamples folders
- Remove the old ARDK package and don’t overwrite old files with new ones, as Unity may not overwrite some of them properly.
- Install the new ARDK package using the guide above
- Add the saved Assets/ARDK or Assets/ARDKExamples folders you copied from the removed ARDK version.
- Read the new ARDK release notes, especially for key changes or upgrades.
- Make any necessary code changes, and rebuild and test wherever you see the need.
Step #5: Add Your API Token for Authentication
This is where we’ll use the ARDK token we generated when setting up our project in our Lightship developer account. We’ll need that token to initiate the networking and awareness features of the ARDK.
You’ll use the ArdkAuthConfig asset from the ARDK to set the API token for API calls.
Here’s how you’ll do it:
Create a Resource folder inside your project asset directory in your ARDK asset. The right-click on ARDKExamples or the asset to which you want to add the authentication assets, go to Create, and click Folder.
Name the folder Resource.
Click on the Resource folder to open it, right-click, go to Create, and add an ArdkAuthConfig asset to that Resource folder. Keep the default name. Don’t change it.
Now you should see your ArdkAuthConfig asset in your Resource folder.
Click the ArdkAuthConfig and go to the Inspector tab.
Go to your Lightship Developer account. Then, in your Project Overview, click the copy icon under API Key to copy that APK key you’d generated.
Enter that API key into the API Key field you’d see in the Inspector, under the ArdkAuthConfig.
Restart Unity to ensure your app’s key is the correct one.
Avoid sharing your ArdkAuthConfig publicly since it’s saved in your project API key. If you want to share your project in a public Github repository, clear the API key field first or remove the ArdkAuthConfig before you share it.
How to Verify API Key Up and Validity
Launch the LowLevelNetworking scene inside of the ARDKExamples in your Unity project.
Click on the ExampleManager object in the scene hierarchy.
You’ll find the Network Session Manager component in the Inspector tab. Alter the RuntimeEnvironment property to LiveDevice.
Get in the play mode. Next, Click the InIt and Connect buttons in the Game View window. You see “Connected as Host” for successful API token authentication and “Connection Failed” if unsuccessful.
Unsuccessful connections will show either 700 as an error code if your API key is invalid or 701 if you didn’t add the API key correctly in ArdkAuthConfig.
Synchronize Information from a Real-World Camera to Unity’s Camera
For ARDK features to synchronize the information from a device’s real-world camera to the Unity camera, you’ll need a prefab component. The ARDK’s ARSceneCamera prefab would do it. Alternatively, add the ARRenderingManager and ARCameraPositionHelper to your current scene camera.
Follow these steps to add a prefab component to your ARDK.
Step #1: Delete all unnecessary default cameras
Step #2: Navigate to the Project View and ARDK > Extensions > Prefabs.
Step #3: Bring ARSceneCamera into the scene and position it to your preference
If you add any other ARDK managers to the scene, ensure to set any camera field they might have to this camera.
Enable Awareness Features
You’ll need to create ARSession to enable awareness features.
An ARSession does three things:
- Configure the AR properties you want to enable
- Give access to the AR system’s present world state
- Allows users to subscribe to AR events
See this resource for how to create an ARSession.
You can also follow the video work-through here:
Video Place Holder for the Video Linked Here
Connect to Niantic’s community to speed up your learning. Niantic is investing heavily in the community and holds three different types of frequent live events and webinars:
- Product Session: a deep dive on certain functionality of Niantic lightship
- Brown Bag Sessions: a non-technical session for beginner developers
- Dev Days: two-hour-long developer workshops to gain build up your experience. It includes introductory product-based workshops.
You can access the community right from your Lightship developer account.
Click the Join the Conversation button.
Join the Lightship community to access community support and the latest updates on the platform's news and events. And you can participate in the community in three ways:
- Forum: Get real-time community support for your projects from community members
- Blog: Follow Niantic lab’s updates, events, news, and developments
- Showcase: Find and follow other developers’ Lightship projects to discover what’s possible and how they’re making money from their work
Some of the key forum sections include:
- I’m Stuck
- Bug reports
- Feature requests
- Introduce yourself
- Show off
Users get a healthy mix of socializing with other developers, co-creating the platform by requesting features, inspiring others by showing off their work and fixing bugs.
You’ll use one login information for your Lightship developer login and community access.
You can learn about Niantic’s events on their events page. In addition, they recently released the Wayfarer app to work with developers in mapping the physical world.
Niantic also organizes physical meet-ups in cities where the visual positioning system (VPS) is gaining the most usage. The first cities on the list include Seattle, London, and Los Angeles.
Follow their website to learn more!
Download the XR Development with Unity Syllabus